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In This Issue 


Conducting a survey using traditional telephone methods is relatively inexpensive and 
maintains direct, albeit remote, contact between interviewer and respondent. These two 
characteristics of telephone surveys - low cost and human contact - explain to a large extent 
why such surveys became popular. The extensive research that took place during the last decade 
is summarized in the book Telephone Survey Methodology edited by R. M. Groves et al. (1988) 
and in Volume 4, Number 4 of the Journal of Official Statistics. This issue’s special section, 
devoted to data collection and capture, presents further developments, with the telephone playing 
a major role in both old and new ways. 

In the first paper of the special section, Werking and Clayton look at research on telephone 
collection methods done during the past seven years at the U.S. Bureau of Labor Statistics (BLS). 
They show how the research has led to the decision to implement computer assisted telephone 
interviewing (CATI) and touchtone data entry (TDE) in the Current Employment Statistics Survey 
during 1991 and 1992. The authors also mention a new technology, voice recognition, as an 
alternative to TDE. 

Phipps and Tupek discuss one particular area of the above research at the BLS, namely, 
measurement errors in TDE surveys. They conclude that entering extra or incorrect digits has 
the most impact, but that this can be reduced via editing, particularly when longitudinal data 
are available. They also suggest ways of improving TDE systems. 

The Mitofsky-Waksberg random digit dialing (RDD) procedure is a commonly-used method 
for selecting households. Brick and Waksberg discuss an inefficient operational feature of the 
procedure and propose a modified method. They study its statistical properties and provide 
guidelines for choosing between the original and modified variants of the procedure. 

Bethlehem and Keller discuss Blaise, an innovative software package developed at the 
Netherlands Bureau of Statistics. They show how Blaise is used as a tool to integrate various 
steps in survey processing, including data collection and data entry. As the use of portable 
computers for sample surveys increases, tools such as Blaise will become indispensable. 

Like most statistical agencies throughout the world, Statistics Canada has studied increased 
and improved telephone usage for survey taking. Recent research and testing for household 
surveys is summarized by Drew in the last paper of the special section. The implications of the 
results for the redesign of the Labour Force Survey are also presented. 

For sampling on successive occasions, a normal linear regression model can be used. Bellhouse 
obtains the marginal and conditional likelihoods for the correlation matrix of this model. 
Applications of these likelihood methods are given, both for the case of simple random sampling 
and for more complex designs. 

Schiopu-Kratina and Srinath describe the methodology of the Survey of Employment, Payroll 
and Hours (SEPH) conducted by Statistics Canada. SEPH is a large monthly establishment 
survey which uses a rotating sample. The determination of monthly sample size is quite complex 
due to continuous changes in the population. Some possible simplifications of the design are 
mentioned. 

Kott shows how some techniques from design-based sample survey theory, namely the 
inclusion of sampling weights and the mean square error (mse) estimator based on linearization, 
can be used in the estimation of a system of linear equations. He shows that the use of sampling 
weights may be preferable when the existence of missing regressors is likely. Furthermore, the 
mse estimator based on linearization is nearly unbiased for many error structures. 


2 In This Issue 


The need to make information available quickly, combined with the need for accurate estimates 
based on all available data, leads to a process of revision of the national accounts estimates. 
Biggeri and Trivellato review recent developments in the analysis of the reliability of national 
accounts estimates that are subsequently revised. An empirical analysis using data from Canada, 
Italy and the United States is also presented. 


The Editor 


As we were going to press, we learned that Dr. M.N. Murthy, Director of the Applied Statistics 
Centre, Madras, India passed away on 2 April, 1991. Dr. Murthy made important contribu- 
tions to survey methodology and was the author of a well-known textbook, Sampling Theory 
and Methods. During his career he worked at the Indian Statistical Institute and the UNO- 
sponsored Statistical Institute for Asia and the Pacific. He was a Fellow of the ASA, the Royal 
Statistical Society and the International Statistical Institute. We were fortunate to have him serve 
on our editorial board. He will be missed by his colleagues and former students around the world. 
He is survived by his wife Vyjayanthi and daughter Shashi. 
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Enhancing Data Quality Through the Use of 
Mixed Mode Collection 


GEORGE S. WERKING and RICHARD L. CLAYTON! 


ABSTRACT 


A chronic problem in the preparation of time critical estimates is the significant limitations inherent in 
data collection by mail. To address this issue, the U.S. Bureau of Labor Statistics has conducted an 
extensive 7 year research effort into the use of computer assisted telephone interviewing (CATI) and the 
computer assisted self interviewing (CASI) methods of touchtone and voice recognition self-response. 
This paper will summarize some of the significant results of this research covering both performance 
and cost data. The paper will conclude with a discussion of a large scale implementation program of these 
techniques for a monthly sample of 350,000 establishments. 


KEY WORDS: Employment statistics; Revisions; CATI; Touchtone collection; Voice collection; Cost 
analysis. 


1. INTRODUCTION 


1.1 Employment Statistics in the United States 


On the first Friday of each month, the U.S. Bureau of Labor Statistics releases data on the 
United States’ employment situation for the previous month. On release day, the Commissioner 
of Labor Statistics appears before the Joint Economic Committee of Congress and provides 
a detailed analysis of the current month’s data and trends; at the same time, the data are made 
available to the news media and the financial and business communities. This closely watched 
set of statistics is the earliest indicator available on the previous month’s economic activity 
and is used as a major gauge of the health of the U.S. economy. The data in the release cover 
employment, hours and earnings by detailed industry which are derived from the Bureau’s 
350,000 unit monthly establishment survey - the Current Employment Statistics (CES) survey - 
along with labor force and unemployment data which are derived from the Bureau’s 60,000 
unit household survey - the Current Population Survey (CPS). 

The establishment survey data have many important economic uses. Due to the CES survey’s 
size and timeliness in conjunction with the importance of the basic payroll statistics which it 
collects, the CES monthly estimates are not only used as principal economic indicators by 
themselves but they are also included in the development of many of the Nation’s other major 
economic indicators including: Personal Income for the Gross National Product, the Index 
of Leading Economic Indicators, the Index of Coincident Indicators, the Industrial Produc- 
tion Index, Real Earnings measures and Productivity measures. While the timeliness and 
accuracy of the CES statistics are essential in analyzing the current economic conditions in the 
United States, the CES survey has had to rely on a mail data collection process since its incep- 
tion in the early 1900s. This collection process results initially in the publication of 
“‘preliminary’’ estimates for higher level aggregates using only the sample returns received to 
date followed by ‘‘final’’ estimates two months later which use the full sample. The process 
of producing both preliminary and final estimates for a given month periodically yields a 
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substantial revision to the initial estimates. These revisions not only affect the basic CES 
statistics but also the other statistics which make use of CES estimates as input. To address 
this issue, the Bureau initiated a research program into automated telephone data collection 
approaches with the objective of substantially reducing the size and frequency of large revisions 
in the preliminary estimates. 

This paper provides an overview of the Bureau’s 7 year research program into automated 
telephone collection techniques and summarizes some of the most significant results. The 
following sections describe the CES survey process; discuss the research program evaluating 
Computer Assisted Telephone Interviewing (CATI), Touchtone Data Entry (TDE) and Voice 
Recognition (VR) data collection methods; detail some of the significant research results 
covering both performance and cost data; and conclude with a discussion of the large scale 
implementation program of these methods in the CES survey. 


1.2 Current Employment Statistics Survey 


The CES survey, with 350,000 units, is the largest monthly sample survey in the United States. 
It is conducted by the Bureau as a Federal-State cooperative program under which the Bureau 
specifies the survey’s sample design and operational procedures while each State conducts all 
data collection and edit reconciliation activities. The Bureau produces and publishes exten- 
sive monthly industry detail at the 2, 3 and 4 digit industry levels for the Nation as a whole 
while each State produces monthly State and area (270 Metropolitan Statistical Areas) estimates. 

The CES estimates are widely regarded as highly accurate economic statistics. Once each 
year, complete (or universe) employment counts for the previous year become available from 
the Unemployment Insurance tax records; these counts are used to annually benchmark (realign) 
the CES sample estimates to these universe counts. The annual benchmark process yields more 
accurate current monthly estimates along with providing an annual estimate of overall survey 
error. The average difference in the CES final sample estimate versus the complete universe 
count over the past 5 years is under 0.2% with 4 years in the 1980s when the difference was 
approximately zero. While the CES final monthly estimates are regarded as highly accurate 
relative to the universe counts; the preliminary monthly estimates, which are based on approx- 
imately 50% of the mail sample returns, have been periodically subject to large revisions when 
compared to the final estimate that is available 2 months later. Over the years, some improvement 
in reducing the size of the monthly revisions has been made; however, periodic large revisions 
have been viewed as a byproduct of conducting a large decentralized mail data collection process. 

The decade of the 1980s brought about a number of changes for the CES program which 
would significantly alter the urgency and options for resolving the monthly revision issue. The 
1980s created a far more quality conscious user constituency and while the CES products had 
not necessarily deteriorated, the CES users’ expectations on quality and ‘‘fitness for use’’ had 
greatly increased. Much of this new way of thinking is directly attributable to the efforts of 
Deming, Juran and others on the subject of quality management. The 1980s also saw a much 
greater focus on the uses and the importance of the CES payroll statistics in assessing the current 
health of the U.S. economy, but with the rise in the use and the visibility of the CES statistics 
came a corresponding user frustration with monthly revisions. The 1980s also ushered in some 
dramatic new technological breakthroughs, most notably in microcomputers. This new 
technology offered survey agencies many new opportunities for improving data collection 
control and quality that included: Computer Assisted Telephone Interviewing, Touchtone Data 
Entry, Voice Recognition, Computer Assisted Personal Interviewing and FAX. Several of these 
methods would ultimately offer options to significantly improve timeliness and quality at an 
equivalent or reduced ongoing program cost. 
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The 1980s saw the Bureau shift from experimental research in the CES survey to full 
production testing of some of the most advanced state-of-the-art automated collection tech- 
niques then available, with major implementation of these techniques scheduled for 1991. 


2. CES RESEARCH PROGRAM 


2.1 Research Goals 


In the early 1980s, the Bureau began an extensive 7 year research effort into the causes of 
late response and alternative collection methods which could significantly increase response 
rates for the preliminary estimates. The focus of the survey research centered around obtaining 
answers to three basic questions: 


e Are data available at the establishment in time to respond by the publication deadline for 
the preliminary estimates? 


e Are there data collection methods which can ensure an 80-90% response rate under these 
tight time constraints? 


¢ Can the cost of these data collection methods be controlled at about the same level as the 
current mail collection costs? 


At the conclusion of the research program, a mixed mode CATI/TDE collection approach 
emerged which satisfied the response rate and cost constraints for the survey. The following 
sections provide a brief description of these personal computer (PC) based data collection 
methods, the research tests, the response rate results and the cost analysis. Further details on 
these tests are documented in the research papers listed in the references. Additionally, recent 
results on measurement error for Touchtone collection are presented in a paper by Phipps and 
Tupek, this issue. 


2.2 Data Collection Methods 


The CES survey has a very limited data collection time period available to meet the 
preliminary estimates publication deadline. The CES survey’s reference date is the payroll period 
containing the 12th of the month; thus, there are only 2 1/2 weeks available to collect, keypunch, 
edit, tabulate, validate and publish the data. In order to meet these tight time constraints, a 
collection method must be able to obtain the required data as soon as they become available 
within the establishment. The four data collection methods studied are described in turn below. 


Mail - The CES questionnaire is a single page mail-shuttle form which provides space for the 
employer to record 12 months of data. The employer receives the questionnaire in the mail 
each month on or about the 12th of the month (i.e., the survey reference date) and subsequently 
fills in the row of data items corresponding to the current month. There are five basic data 
items collected: all employees, women worker employment and production (or nonsupervisory) 
worker employment, hours and earnings. Once completed, the employer mails the form back 
to the State agency where it is keypunched and edited. The form is then filed so that it can 
be mailed back to the employer for the collection of the next month’s report. As indicated 
earlier, this process currently yields a 50% response rate in the 2 1/2 weeks available for the 
preliminary estimates. 

Computer Assisted Telephone Interviewing - Under CATI collection, the employer is mailed 
the CES questionnaire once at the beginning of the year and retains it for recording each month’s 
data throughout the year. Each month as the payroll data become available, the employer fills 
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in the data items for that month and waits for the prearranged CATI call from the State agency. 
When the State agency calls, the data are collected under CATI, edited and a time for the next 
month’s collection call is arranged. 


Touchtone Date Entry - Under TDE reporting, the employer does the same activities as under 
CATI except instead of waiting for the State agency’s CATI call, the employer now calls an 
800 telephone number connected to the touchtone PC located at the State agency. The employer 
then touchtone enters the data items following the prompts in the automated CES interview. 
As each data item is entered by the employer, it is read back for respondent verification. 


Voice Recognition - VR data reporting is identical to touchtone collection except the employer 
no longer needs to have a touchtone phone. The employer now reads the data as they appear 
on the form and the voice PC translates and reads back the data to the employer for verification. 
The VR system is speaker independent and accepts continuous speech; it recognizes the digits 
0 through 9 and ‘‘yes’”’ and ‘‘no’’. 


2.3. Research Tests 


The Bureau began developing a PC-based CATI system in 1983 for use in a two State test that 
began in 1984 (Figure 1). The CATI system developed by the University of California at Berkeley 
was selected for the test and was subsequently used throughout the research effort. A small 
random sample of 200 units was selected in each State and collection procedures and systems 
were refined over the next 7 years. The initial research tests were highly successful in the response 
rates they achieved and the tests were expanded to 9 States in 1986 and then to a total of 14 
States in 1988. The composition of the test sample was also changed in 1986. Instead of selecting 
random samples of the full CES sample, the subsequent research tests focused only on random 
samples of habitually late CES respondents (i.e. , those units which had a response rate of under 
20% for the preliminary estimates publication deadline). Thus, the success of the new collection 
methods of CATI and TDE was measured in terms of their ability to move samples of reporting 
units with a 0-20% preliminary estimates response rate to a stable ongoing 80-90% response 
rate. By the end of the CATI research phase in 1990, the Bureau was collecting over 5,000 units 
monthly under CATI and had conducted well over a quarter of a million CATI interviews. 

While CATI was proving to be highly successful in improving response rates, it also became 
clear by 1985 that ongoing CATI collection would be more expensive than the existing mail 
collection. At this time, a separate path of research was begun on how to reduce the cost of 
CATI, while still maintaining the high monthly response rates which it was achieving. While 
improvements were made in reducing the length of time required for a CATI interview, it was 
anew alternative PC-based telephone reporting method which would offer dramatic reductions 
in the collection costs of CATI. 

By 1985, many U.S. banks were operating a version of touchtone entry verification for check 
cashing at drive-in windows. The Bureau identified a PC-based touchtone reporting system 
suitable for survey research testing and by 1986 was conducting a small two State test of this 
technique for collecting data. TDE was not viewed as a direct replacement for mail nor as a 
competitive method to be tested against CATI. CATI’s role was to take habitually late 
responders and turn them into timely responders through personal contact and an educational 
process, while TDE’s role was to take these timely CATI responders and maintain their response 
rates at the same high level, but at a greatly reduced unit cost. Over the 5 years of data collec- 
tion, TDE has also proven to be a very successful and reliable method of telephone data 
collection. The research phase for TDE is now also being concluded with over 5,000 units 
continuing to report monthly under TDE across 14 States; in total the Bureau has collected 
over 100,000 schedules using this new automated reporting method. 
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As a natural follow-up to TDE, the Bureau is currently conducting several small research 
tests of anew Voice Recognition reporting system. Preliminary results for VR reporting have 
replicated the same high monthly response rates achieved under TDE, but with the important 
advantage that respondents find VR reporting more natural and generally prefer it over TDE. 
At this time, the cost of the VR hardware is approximately 15 times that of TDE; however, 
within several years as the initial costs of VR drop, this collection method should become a 
viable replacement for TDE. 


2.4 Research Results 


Over the past 7 years, the Bureau has been able to establish that payroll data is available 
in most firms prior to the publication deadline for the preliminary estimates and that CATI 
collection has the ability to take traditionally late mail responders (i.e. , 0-20% response rate 
for preliminary estimates) and within 6 months turn them into timely responders with response 
rates of 82-84% (Figure 1). These response rates have been remarkably stable over the years 
as the CATI sample has been expanded from 400 units to 5,000 units and the number of 
participating States increased from 2 States to 14 States. The research results indicate that the 
data do exist at most establishments in time to meet the publication deadline and that CATI 
collection can raise mail response rates by 60-80% for these late respondents and this rate can 
be maintained in the targeted range of 80-90% over long periods of time. The principal limiting 
factor in a respondent’s ability to make the publication deadline was found to be the length 
of the firm’s pay period (Figure 3). Employer pay periods are generally weekly, biweekly, semi- 
monthly or monthly. Weekly and semi-monthly payrolls can almost always be collected in time 
for publication with biweekly pay periods available most of the time; however, most monthly 
payroll systems close out well after the publication deadline. Monthly payrolls have been one 
of the largest factors in limiting the CATI response rates to the 82-84% range. 

Several other important results have come out of the CATI research. Under CATI, approx- 
imately 60% of the respondents will have their data available on the prearranged date for the 
first call with the remaining 40% using the first call as a prompt call (Figure 1). This rate has 
varied little across States or over the years of testing. A small test is scheduled to be conducted 
to see if an advance postcard notice to the respondent shortly before the prearranged CATI 
contact date will significantly limit the number of callbacks required. 
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Figure 1. Research Summary 
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The average time for a CATI interview depends on the number of items to be collected, the 
time efficiency of the interview instrument, and the experience of the data collector. The average 
time for a CATI call (Figure 1) was reduced by one-third as the CATI instrument was 
streamlined and interviewers became more experienced. Another very important concern in 
the testing was the effect of CATI on sample attrition. There was some concern that employers 
would not want to be constantly bothered by telephone contacts and would drop out of the 
program. However, the sample attrition rate for CATI was about one-third of that for mail 
with almost no loss of large reporters under CATI. In summary, CATI appears to have come 
close to maximizing the achievable response rate for the preliminary estimates while also 
enjoying broad support from the respondents. 

Due to the increased cost associated with CATI collection, the Bureau initiated research 
into touchtone collection. During the 4 years of testing, TDE has demonstrated the ability to 
take timely CATI reporters having 82-84% response rates and maintain these high rates under 
completely automated TDE reporting (Figure 1). The importance of this result lies in the cost 
savings under TDE collection versus CATI collection. One of the major concerns for TDE 
collection was that, unlike CATI where respondent contacts are scheduled throughout the day, 
TDE respondents might tend to call during the same time period thus generating busy signals 
and require an excessive number of touchtone PCs to handle peak load reporting. Fortunately, 
this was not the case, and while the touchtone PCs are on-line 24 hours a day, most calls are 
relatively uniformly distributed between 8am and 5pm (Figure 4). TDE respondents tend to 
require the same proportion of prompt calls as CATI respondents - approximately 40%. 
Methods are currently also being tested to reduce the TDE prompting workload. One major 
advantage for the respondent is that TDE collection requires only one-half of the time of a 
CATI interview with the average TDE interview lasting only 1 minute and 45 seconds. Addi- 
tionally, touchtone phones are widely available at most establishments; current estimates 
indicate that over 80% of employers could report under touchtone data collection. While TDE 
reporting offers many advantages to the survey agency, its strongest feature is respondent accep- 
tance; respondent reaction to touchtone reporting has been very positive due to its speed and 
convenience for the respondent. 

One general observation concerning the development of a CATI research program is that 
it is not critical which CATI hardware or software system is used during the research phase 
as long as it is reasonably flexible for change. The final results from testing may suggest very 
different CATI requirements for production implementation than those originally required 
for the research program. The most important and time consuming activity is the development 
and refinement of the methods and procedures for respondent contact. Once effective methods 
and procedures are developed, the requirements for the ‘‘right’’ system become more obvious. 


2.5 Cost Analysis 


With the performance testing and respondent acceptance for CATI and TDE proving to 
be highly successful, the final phase of research shifted to analyzing the transitional costs of 
CATI and the ongoing costs of TDE collection. 

The major “‘labor’’ and ‘‘non-labor’’ cost categories were studied for mail, CATI, and TDE 
collection (Figure 2). The study looked not only at estimates of current cost, but also at projected 
costs over the next 10 years using the current rate of increase for the major cost items. Since 
CATI was to play only a 6 month transitional role (i.e., moving late responding mail units to 
on-time responding CATI units) prior to conversion to ongoing TDE, the major focus of the 
cost analysis was on the cost tradeoffs between ongoing mail versus ongoing TDE data 
collection. 
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Figure 2. Data Collection Costs (arrows show direction of recent price change) 


For the labor categories, the monthly mailout, mailback, check-in and forms control opera- 
tions of mail were replaced by a single annual mailout-only operation under TDE, thus 
eliminating a large monthly clerical operation in the States. The batch keypunching, keypunch 
validation, and forms control operations under mail were completely eliminated under TDE, 
where the respondent touchtone enters the individual firm’s data and validates each entry. 
Another major quality and cost-efficiency advantage of TDE was that procedures for telephone 
nonresponse follow-up became far more feasible under TDE than mail. Under TDE, an 
accurate up-to-date list can be generated of respondents who have not yet called in their data, 
this list can then be used to conduct brief telephone prompting calls. Under mail, telephone 
follow-up activities of ‘‘apparent’’ nonrespondents were awkward since the State staff did not 
know whether the respondent’s form was not yet completed, currently in the mail, in the State 
check-in process or at keypunch; in addition, respondents who had recently sent their form 
tended to resent the additional reminder for an activity that they perceived as completed. Due 
to the voluntary nature of the program and the uncertainty of a respondent’s response status, 
telephone prompting under mail was only used for critical (large employers) units. 

There were no significant cost savings made for edit reconciliation as the number of edit 
failures under TDE remained at about the same level as under mail. This was also true for 
postcard reminders where the number of postcards used under mail collection for late 
respondents was approximately the same as the number used under TDE, where respondents 
received an ‘‘advance’’ postcard notice to touchtone their data by the due date. 
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In the non-labor categories, the cost of postage under mail (currently 58 cents per unit) is 
replaced by the cost of a telephone call and the amortized cost of the TDE machine (together 
currently 46 cents per unit). Postage is a continually increasing cost with an annual price increase 
of approximately 5% (Figure 2). The rising cost of postage is driven by annual labor cost 
increases (+ 5.7%) and by fuel costs (also generally increasing) with labor accounting for over 
80% of total postage costs. In contrast, under TDE the cost of telephone calls has been 
decreasing in recent years (— 1.7%), along with the cost of microcomputers (— 19.5%). 

Excluding the additional new requirement of full telephone nonresponse prompting for TDE, 
there are demonstrable cost savings in shifting from mail to TDE. Perhaps more importantly, 
under a 10 year projection of future costs of these two collection methods (Figure 5), these 
savings grow substantially. Attempts will be made to redirect future cost savings from TDE 
to help offset the full nonresponse prompting activities. 

There are several major conclusions concerning TDE reporting which have emerged from 
the performance and cost analysis review. (1) The traditional view that mail is the least expen- 
sive collection option available to statistical agencies is no longer true. The major technological 
breakthroughs of the 1980s in automated telephone collection can not only reduce the collection 
cost below that of mail but can also improve timeliness and control over the collection process. 
Additionally, over the next 5-10 years, the cost of mail will become even less cost competitive 
with these high-technology/low-labor collection approaches due to the increasing labor and 
postage costs associated with mail. (2) The transition of respondents from mail to TDE appears 
to cause very little disruption to monthly reporting. In the Bureau’s follow-up interviews with 
respondents who were converted to TDE, results have shown that respondents have very little 
trouble adapting to this new method of reporting. Virtually all respondents completed their 
first month TDE report accurately and without assistance, with many respondents commenting 
on the ease of reporting under TDE. (3) TDE can be viewed as a reliable replacement method 
for survey data collection; over the past 4 years of collection there have been no major equip- 
ment failure problems or disruptions of the collection process. Minor equipment problems have 
been easily resolved using a back-up PC when required. In addition, future back-up protection 
for the State TDE collection process will involve the use of a call forwarding option to reroute 
calls to a central site should major problems occur at the State. 


3. IMPLEMENTATION 


3.1 Major Issues 


By the end of 1989, the Bureau had completed a very successful research program and had 
sustained the high performance levels over 7 years. However, there is a significant difference 
between the completion of successful research and full scale implementation of new methods. 
While over 10,000 units were being actively collected under these new techniques, these units 
represented under 3% of the CES sample. Proposed collection changes for a monthly sample 
of 350,000 units which has been collected for well over half a century under a decentralized 
State mail collection environment requires not only a very strong demonstrable user need but 
also broad-based support at national, regional and State levels. 

As it turned out, the user need had begun to change in the early 1980s. During the 1980s, 
the U.S. economy experienced the longest sustained peace-time growth period in its history, 
with over 19 million jobs created, and unemployment rates at their lowest levels since the early 
1970s. By the mid 1980s, economic policy was firmly focused on establishing non-inflationary 
economic growth. The monthly CES employment growth and wage data were being closely 
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monitored for signs of wage-induced inflationary pressures resulting from strong job growth 
during a period of low unemployment. With this greatly increased use and visibility of the mon- 
thly data, came a corresponding user frustration with the periodic large revisions to the 
preliminary estimates. 

While monthly revisions to the preliminary estimates had always been a part of the CES 
survey process and even though the size of these revisions had been reduced over the years, 
large revisions of over 100,000 in the preliminary monthly employment estimates of over-the- 
month change were now being viewed as unacceptable. These user demands for greater accuracy 
in the preliminary estimates would lead the Bureau to develop proposals for the implementation 
of automated CATI and TDE collection methods into the U.S. government’s largest monthly 
survey. While user demand is critical, major changes of this magnitude could not be under- 
taken without full support at the State level where data collection actually occurs. One guideline 
which remained constant throughout the research program was that the collection system was 
ultimately the States’ data collection system and therefore must be designed to integrate well 
into their survey environment and create as minimal an organizational impact as possible. To 
that end, the CATI and TDE systems remained open for change throughout the research 
program. As many State suggestions and requirements as possible were taken into account with 
each new release of the systems. The success of much of the development work can be credited 
to the resiliency and endurance of the 14 research States as they made constant recommendations 
for improvements in systems and procedures. In the end, the CATI interview instrument had 
moved from an awkward simulated household survey type interview approach to a fast and 
efficient ‘‘screens’’ and ‘‘windows’’ approach well suited for capturing and editing longitudinal 
economic data. Thus, at the conclusion of the research phase, the systems and procedures were 
well tested and refined across a wide range of States. This approach to testing brought with 
it a strong sense of confidence in the methods and the systems at the State level. This would 
prove to be essential for the Bureau’s proposed quick production implementation timetable 
of these state-of-the-art collection methods. 


3.2 Approach and Impact 


The main focus of the implementation proposal was the control of revisions in the 
preliminary estimates. Over the past 5 years, approximately 40% of all revisions were over 
50,000 with 13% of the revisions exceeding 100,000 (i.e., large revisions) (Figure 6). The goal 
of the implementation study was to identify a minimum set of late responders which, if obtained 
by the publication deadline, would control the size of revisions to what was considered to be 
in an acceptable level (under 50,000 revision in the over-the-month employment change). While 
one obvious approach was to convert all 175,000 late respondents to the new collection methods, 
this approach was considered to be lengthy and costly. While there was a need to responsibly 
control the size of revisions (i.e., not necessarily completely eliminate all revisions), there was 
also a corresponding need to resolve this problem in as timely a fashion as possible (i.e., convert 
the smallest number of units necessary to control revisions to under the 50,000 level). 

Establishment surveys, unlike household surveys, generally have differential weighting for 
individual units with very large units being ‘‘certainty’’ units in the sample design. In the CES 
sample design, units with 100 + employees make up only 20% of the sample (i.e. , 75,000 units), 
but account for over 83% of the unweighted sample employment. These units tend to have 
a much lower response rate for the preliminary estimates so that if the late respondents’ 
employment trend differs from the early respondents, these units can create a substantial 
revision in the sample estimates. Revision impact studies were conducted to assess the affect 
of large employers 100+ on the preliminary estimates. To test the impact of large employers, 
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late respondents in size class 100 + were included in the original sample used for the preliminary 
estimates and the estimates were recalculated. These new estimates were then compared to the 
original preliminary estimates to determine the impact of 100+ employers on revisions. The 
results indicated that from one-half to two-thirds of the revision was attributable to these units. 
These studies were repeated over several months with similar results. Applying these projected 
reduction rates in revision size to revisions over the past 5 years resulted in over 97% of all 
revisions being below the 50,000 level as compared to the current level of only 60%. This greatly 
reduced targeted sample size for conversion to CATI/TDE provided for an accelerated 
implementation time schedule consistent with controlling conversion costs to the minimum 
level necessary to protect against large revisions in the preliminary estimates. The Bureau will 
begin the implementation of CATI/TDE collection methods in 25 States in 1991 with planned 
expansion to all States in 1992. With the implementation of the new collection methods, the 
Bureau will be able to help resolve one of the most difficult and visible quality issues affecting 
the CES user community. 


4. SUMMARY 


The decade of the 1980s has brought about many changes for survey agencies. Some of the 
changes can be viewed in terms of our accomplishments made over the decade while others 
are more subtle and need to be viewed in terms of the changes in the survey environment in 
which we operate. 

The 1980s created a far more quality conscious user constituency which is quick to identify 
and point out our product limitations. While our products may not have deteriorated, our users’ 
expectations on quality and ‘‘fitness for use’’ have greatly increased. This is an issue which 
we as Statistical agencies must be able to respond to in order to maintain our credibility with 
the user community. The 1980s also ushered in dramatic new technological breakthroughs most 
notably in microcomputers. The new technology has offered survey agencies many new oppor- 
tunities for improving data collection control and quality including: CATI, CAPI, TDE, VR 
and FAX. Some of these options offer improved quality and control at lower ongoing costs. 
The decade of the 1990s may well offer even greater opportunities for using technology to 
improve our data collection timeliness and quality at lower costs. 

As we look at the status of our statistical programs, we often find very rigid environments. 
The data collection approach for our surveys often date back to their inception. Our data 
collection cost assumptions and cost studies are usually well outdated and often simplistic in 
approach. Since data collection generally represents the largest part of a survey’s cost, it is 
usually well entrenched in the agency’s organizational structure and can be quite difficult to 
restructure in order to accommodate large scale change. It is within this survey environment 
that we will face the major challenges and opportunities of the 1990s. 


The challenge for statistical agencies in the 1990s will be threefold: 
© to be responsive to the changing quality needs of our users; 


© to attempt to have our research stay up with the rapid change of technology and automated 
data collection approaches; and perhaps more importantly 


© to continue to find ways to incorporate successful research into our ongoing programs. 


These challenges will determine the cost and quality competitiveness of our programs and our 
agencies in the future. 
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Assessing Measurement Errors in a Touchtone 
Recognition Survey 
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ABSTRACT 


Electronic data collection utilizing touchtone recognition is in place for a monthly establishment survey 
at the Bureau of Labor Statistics. The Touchtone Data Entry (TDE) system features digitized phrases 
requesting respondents to answer questions using the numeric keypad of a touchtone telephone. TDE 
has substantial implications for lowering survey costs; many labor intensive activities are eliminated. 
However, little is known about measurement errors associated with this mode of data collection. This 
study assesses TDE mode error using three sources of data, which allow for analyses of errors associated 
with selected aspects of the human-machine interface. In addition, instrument design issues associated 
with mode error are addressed. We conclude by extending the implications of our findings to other surveys. 


KEY WORDS: Mode of data collection; Human-machine interface; Computer-assisted self interviewing. 


1. INTRODUCTION 


The U.S. Bureau of Labor Statistics (BLS) issues monthly employment estimates for the 
United States from a survey of 350,000 business establishments. This survey, the Current 
Employment Statistics (CES) survey, provides one of the earliest monthly measures of U.S. 
economic health. However, the preliminary estimates from the survey are released with data 
from only about one-half of the business establishments in the survey. Revised estimates are 
produced two months after the initial press release. The low response rate for the initial press 
release can result in large revisions to the estimates. The BLS began investigating the use of 
automated collection techniques in 1983 to increase the timeliness of response and reduce the 
potential for large revisions. 

The CES survey has traditionally been collected by mail through state employment security 
agencies. Research tests conducted between 1984 and 1986, involving the replacement of mail 
collection with computer-assisted telephone interviewing (CATI), have shown CATI to be an 
effective means for improving the timeliness of response (Werking, Tupek, Ponikowski and 
Rosen 1986). Average response rates under CATI collection have been between 85 and 90 
percent for preliminary estimates, compared to 45 to 50 percent with mail collection. While 
CATI collection has been effective in improving the timeliness of response, the cost of full CATI 
collection in the CES survey cannot be absorbed within the survey’s current budget. Research 
has been conducted since 1986 on touchtone data entry (TDE) to develop an alternative col- 
lection method with the performance gains of CATI, but with a lower unit cost (Ponikowski, 
Copeland and Meily 1989). Further discussion on the use of CATI and TDE collection for the 
CES survey is provided in the paper by Werking and Clayton, this issue. 

Recent tests of the TDE system provide data on the timeliness of response, the cost of 
collection, and edit failure rates. BLS tests show that TDE collection with CATI back-up is 
as timely and effective as CATI collection (Werking, Tupek and Clayton 1988). In addition, 
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TDE lessens survey costs considerably. Current cost estimates indicate that the monthly unit 
costs for TDE are approximately 30 percent less expensive than mail collection, while the 
monthly unit costs for CATI collection are 20 percent more expensive than mail (Clayton and 
Harrell 1990). Collection by TDE is now expanding in the CES, beginning with the establish- 
ments with the greatest employment. Research continues in determining ways to improve the 
touchtone system, even though respondents’ acceptance of touchtone collection is highly 
favorable. 

The purpose of the current research is to identify respondents’ problems using the touchtone 
system and to measure errors associated with the mode. Results indicate where improvements 
are needed to reduce respondent problems as well as errors. Section 2 provides background 
information on TDE and the operations of the BLS TDE system. In the third section we address 
the potential errors due to the TDE mode of data collection, which requires a human-machine 
interface. In section 4 we describe the three sources of data collected for the study. A record- 
check survey, machine-recorded data and a respondent debriefing survey are used in the analysis 
of problems and errors. Section 5 includes the methods, analyses and findings from each of 
the data sources. Finally, in section 6 we provide an overall assessment of measurement errors 
due to this mode of collection, suggestions for improving the system, and implications for other 
surveys. 


2. TOUCHTONE DATA ENTRY 


The primary reason to consider collection by touchtone self-response is to reduce the cost 
of collecting data by CATI, while maintaining the timeliness and quality of CATI collection. 
The CES survey seemed to be a good candidate for touchtone collection, since only five or 
six numeric data items are collected each month. The data items include: all employees, women 
workers, production workers, production-worker payroll, production-worker hours, and for 
some industries, over-time hours or employee commissions. Establishments are asked to report 
totals for each data item for the pay period which includes the 12th of the month. TDE 
respondents are also asked to report their establishment identification number and the month 
for which they are providing data. 


The features of the BLS touchtone system include the ability to: 


- detect legitimate establishment respondents based on a match to a file of establishment 
numbers; 


- vary the set of questions depending on the industry of the establishment; 


- read back all responses for respondent confirmation using a computer simulated (digitized) 
voice (respondents are requested to enter ‘‘1’’ to confirm their answer or ‘‘0’’ to reenter their 
answer); 


- wait two seconds for respondents to begin entering their answer, and wait two seconds 
between digits before interpreting data entry as complete (data entry is also assumed com- 
plete if the entire field length is filled - example: the data item ‘‘month”’’ has a two-digit field); 


— repeat each data item question up to three times (for identification number, month and all 
employees) or request the respondent to confirm that they have no answer for the question 
(for all other data items), if a respondent does not confirm their answer or if no answer is 
provided in the two seconds after the question is read; 


- store the date, start and end time of each call, and all data items (Werking ef a/. 1988). 
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Respondents are mailed instructions on touchtone data entry. The instructions give direction 
on how to use the system and examples of the computer and respondent interaction, such as: 


Computer: Your Response: 
Enter all employees For 25 employees, press 2 and 5 
You entered 2, 5 Press 1 to confirm, 0 to reenter. 


The instructions also include the optional use of the ‘‘#’’ sign to indicate completion of data 
entry for an item. The use of the ‘‘#’’ sign reduces the time of the interview. In addition, before 
reporting their data respondents can call in to try out the system using a special test identification 
number. 

Touchtone respondents are contacted by a telephone interviewer during their first month 
on the system to determine any problems they may have and to provide guidance, if necessary. 
Respondents receive a postcard reminder each month and a prompt call if they do not self- 
report by a specified date. The prompt call asks the respondent to telephone into the touchtone 
system as soon as possible. Data are not usually collected during the prompt call. 


3. MEASUREMENT ERROR IN A HUMAN-MACHINE INTERFACE 


Respondent use of a system such as TDE to answer survey questions has little precedent. 
However, touchtone recognition is widely used in such procedures as electronic banking and 
customer-controlled telephone services. While these services may save time and expense, they 
have a potential to alienate users. Problems and errors can originate with the system, task, 
or respondent. System problems primarily generate nonresponse error, while measurement error 
is related to the task and respondent performance. 

While not directly related to surveys, the human-factors literature suggests several inter- 
related factors that may contribute to performance errors in a human-machine interface. First, 
respondents may not be familiar or comfortable with the technology. Waterworth (1984) sug- 
gests that the language of the human-machine interface is different than human communication 
as actions are performed in an order reflecting computer program logic. Since the ability to 
think in a way that parallels the logic is not a minor exercise, those with limited experience 
may have difficulty understanding the task and using the system. Second, synthetic speech is 
more difficult to understand than natural speech and places greater processing demands on 
working memory (Schwab, Nusbaum and Pisoni 1985). Thus, comprehension and memory 
problems associated with the mode may cause errors. 

Synthetic speech includes both digitized speech, where a human voice is sampled, digitally 
encoded, and stored, and rule-based synthesized speech, generated using text as input (Marics 
and Williges 1988). The TDE system utilizes digitized speech, which is less difficult to under- 
stand than rule-based synthetic speech. However, comprehension problems occur with digit- 
izing, as it introduces distortion into original speech (Cox and Coope 1981). Research shows 
that the understanding of synthetic speech may improve with training. In an experiment on 
perception of synthetic speech, Schwab and colleagues (1985) found that training with synthetic 
speech increases perception performance. Thus, comprehension may improve with exposure 
to and experience with the system. Another factor that may affect comprehension is the pace 
of the system. Marics and Williges (1988) found that the rate of the synthesized speech 
significantly affects speech intelligibility, as measured by transcription errors and response 
latency. However, subjects who received contextual information prior to listening to the speech 
had fewer transcription errors. 
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Thus, potential errors in the human-machine interface can occur from lack of experience 
with the technology and task, and from comprehension and memory problems associated with 
voice clarity and pace. Yet these problems are surmountable, as the evidence indicates that 
experience and training can improve performance. 


4. DATA 


There were several objectives we considered in measuring TDE problems and mode error, 
and determining what data to use or collect. First, it was necessary to identify if and where 
problems were occurring. Second, we felt respondents should identify and interpret problems, 
but we also wanted measures independent of respondent assessment. Third, we needed to 
address problems and errors associated with the task and comprehension, including the possible 
improvement of respondent performance over time. 

We decided to assess TDE problems and mode error using three different data sources, which 
have in common approximately 465 Pennsylvania business establishments. These establishments 
reported their monthly survey data by TDE to the Automated Collection Techniques (ACT) 
Laboratory at the BLS national office in Washington, D.C. A small number of the 
establishments began reporting by TDE to the ACT Lab in April, 1989. Others were added 
monthly through November of 1989. Most of the establishments continued reporting to the 
ACT Lab through April of 1990. The majority of these establishments moved from mail to 
TDE reporting. 

The first source of data has two components. One is the TDE data recorded by machine 
from April to December, 1989. The other component is the same data recorded by 
establishments on a survey form. All respondents receive a yearly survey form on which they 
are requested to record their data for each month. Mail respondents fill in the form each month 
and mail it to the state employment security agency. The agency records the data, then returns 
the form by mail for next month’s collection. CATI and TDE respondents are sent the survey 
form, but they do not return it. However, we sent a request to the TDE respondents to return 
their 1989 survey form, and obtained a 96 percent return rate. We then compared the TDE 
and form data, identifying discrepancies between the two. The TDE and survey-form data 
includes 1,930 observations across a nine-month period. Since establishments were phased into 
TDE slowly, the number of observations per establishment varies. The data cover approximately 
75 establishments for 6-9 months, 200 establishments for 4-6 months and 190 establishments 
for 2-3 months. We refer to these data as the record-check data. 

The second source of data includes machine-recorded information on respondent performance 
during the TDE telephone call. The TDE instrument was reprogrammed in January 1990 to 
automatically count and record the number of times a question was repeated due to nonresponse 
(question repeat), the number of times a respondent reentered data (data reentry) for each ques- 
tion, and the number of times an establishment called and hung up before entering data. Unfor- 
tunately, only the questions asking for the month and all employees total could be explicitly 
separated into question repeat and data reentry. For the data items including women and 
production workers, payroll and hours, we had to combine repeats and reentrys, due to the 
structure of the original computer program. We refer to these data as the machine-recorded data. 

The third source of data is a telephone debriefing survey, conducted from January to April 
of 1990 with the Pennsylvania establishments on their experiences with the TDE system. 
Approximately 411 business establishment respondents completed the interview, an 88 percent 
response rate. The questions covered such topics as voice quality, pace of interview, task 
problems, use of systems features, adequacy of instructional materials, and a system rating. 
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5. RESULTS 


5.1 Record-Check Data 


When we requested TDE establishments to return their survey forms, our first question was: 
how many respondents really used the forms? We speculated that one source of mode error 
was respondents who did not complete the form for use when entering their TDE data, which 
would increase demands on their memory. Those who did not complete the form might be more 
likely to enter and/or verify incorrect data. Thus, the request for the survey forms indicated 
that respondents were to return the form regardless of whether they completed it or not. 
However, of the 96 percent that returned their survey forms, only one establishment mailed 
in a blank form; all others sent in completed forms. While nonrespondents may work from 
memory, most of the respondents had completed their forms, giving us reason to believe 
memory problems due to lack of form use were not a major source of errors. 

When comparing the data received by TDE with that on survey forms, we identified and 
coded discrepancies. The data on the survey form are those we would have received and used 
if the respondents were reporting by mail. The results are shown in Table 1. The first type of 
discrepancy occurred when the TDE data indicated there was no response for a data item, but 
there was a response on the establishment’s survey form. This item nonresponse accounted 
for the greatest number of discrepancies, 82 out of 177, and was quite evenly spread across 
the applicable data items. 

There was a pattern to the item nonresponse by month and establishment. The item 
nonresponse rate was 40 percent higher in the first month an establishment reported by TDE. 
In addition, some establishments had more difficulty than others, indicated by two or more 
nonresponses. Nearly half of all item nonresponse occurred in 18 establishments at or close 
to the time they began responding by TDE. This indicated problems existed with first-time 
use of TDE that might decrease with experience. Since the problem was concentrated in a 
small group, we believe it reflected lack of familiarity with automated processes. The remaining 
item nonresponse had no identifiable patterns; our suspicion was that some establishments 
simply missed the item, possibly due to office distractions, and continued on with the next 
question. 


Table 1 


Record-Check Data - Number and Type 
of TDE Discrepancies 


TDE item nonresponse 82 
1-2 few/too many digits 18 
Slipped on keypad 17 
Dis/confirm - ‘‘1’’, ‘‘0’’ error 14 
Form corrected, not TDE 172 
No apparent error reason 26 
Other reasons 8 


Total iy 
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The second type of discrepancy was entering extra digits or, in a few cases, entering too few 
digits, which accounted for 18 of the 177 errors. This was specifically a problem associated 
with entering the payroll data item, where four respondents tried to enter cents instead of 
rounding to the nearest dollar. Several of the same respondents appeared to enter a half hour, 
50, for production-worker hours rather than rounding. In the third type of discrepancy, the 
TDE numbers were nearly the same as those on the form, but one number off. The number 
entered incorrectly indicated a potential task problem in that the respondent may have had 
their fingers slide over on the keypad to the number directly on the side or below the correct 
digit. This accounted for 17 discrepancies. The fourth type of discrepancy occurred primarily 
for the all employee data item. There were eight establishments who had a ‘‘1”’ entered for 
this item in the TDE data, but had a larger employment number on their survey form. We 
speculate that respondents entered ‘‘1’’ twice when confirming the previous question on month. 

Finally, there were a few respondents who had corrected data on their survey form, but not 
on TDE. There were other discrepancies which we could not explain. In addition, several 
respondents transposed their numbers or were off one category, accounting for the ‘‘other’’ 
reasons. For most of the errors, it was difficult to specifically ascertain if they were caused 
at the time of data entry, or not clearly comprehending the question or numbers being read 
back for verification. We suspected the former, but only for the second discrepancy, adding 
too many digits, could we really rule out comprehension problems. 

The error rates for the survey items, ranging from 1.2 to 2.5 percent, are shown in Table 2. 
The all-employee, women-worker and production-worker questions have a lower percentage 
of errors than payroll and hours. This is not surprising since payroll and hours worked are 
usually four to six digits, compared to two to three digits for the other items. Thus, longer 
strings of numbers cause more difficulty. This may be related to difficulties entering the data, 
lack of respondent motivation in correction, or problems remembering longer strings of 
numbers during validation. 

Table 3 shows the potential effect of the discrepancies on the CES data items, calculated 
by taking the sum of the difference between the values in the TDE system and the form, then 
dividing by the sum of the values on the form. The CES Survey uses a link-relative estimator 
for published estimates. The estimates in Table 3 do not take into consideration this estimator. 
However, the estimates in Table 3 provide an indirect measure of TDE mode error on survey 
estimates. None of the error is significantly different from zero at the five percent level. 
However, the potential for mode error appears to be more serious for production workers, 
payroll and production-workers hours. In this study, the number of production workers are 
overestimated by 7.3 percent, payroll by 7.3 percent, and hours by 4.4 percent. 

Nearly all the discrepancies would have failed the edit parameters used in the CES survey 
and been corrected. The resultant effect of the discrepancies after edit corrections is zero for 


Table 2 
Record-Check Data - Number of Discrepancies and Percent Error by Data Item 


All Women _— Production Purell Production Total 
Employees Workers Workers y Hours 
Discrepancies 23 29 28 48 49 177 
% Error 12 Le 5 2 Pie 1.8 
(SE) (.2) (.3) (.3) (.4) (.4) (.3) 


(N = 1,930 for each item). 
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Table 3 
Record-Check Data - TDE Mode Error for Data Items Before and After Edit Corrections 
All Women Production Ba roll Production 
Employees Workers Workers y Hours 
% Mode error, before 
edit corrections 0.0 0.5 7.3 7.3 4.4 
(SE) (.4) (.3) (5.2) (3.8) (3.7) 
% Mode error, after 
edit corrections 0.0 0.0 0.0 0.0 0.0 


(SE) (0.0) (0.0) (0.0) (0.0) (0.0) 
(N = 1,886-1,930 for each item). 


all data items, as shown in Table 3. Examples of the large discrepancies include a respondent 
who incorrectly entered payroll for the number of production-workers, increasing production 
workers by over ten thousand, and two respondents who incorrectly entered the number of 
production-worker hours for number of production workers, raising the latter by several thou- 
sand. Payroll and hours have similar gross discrepancies, including the respondents who put 
in cents, instead of rounding for payroll. 


5.2 Machine-Recorded Data 


The touchtone system provides a tool for assessing difficulties which respondents have with 
this mode of collection. TDE can record the number of times respondents reenter their data, 
and how often the question is read back a second or third time before they respond. Information 
can also be kept on those respondents who hang-up before entering data. 

The machine-recorded data were collected for a total sample of 1,203 observations over a 
three-month period in 1990. There were approximately 474 unique respondents, many of whom 
provided data for two or three months. There were few differences in the machine-recorded 
data by month, so all data are presented for the three months combined. 

Figure | provides, for each data item, the percent of calls for which the question was stated 
to the respondent more than once. The question could be stated a second time if the respondent 
does not answer in two seconds (repeat), or if the respondent fails to confirm his or her answer 
by entering ‘‘1,’’ after it is read back (reenter). The figure indicates that the first two questions 
on the month and all employees, and the payroll and hours questions have higher rates of 
repeating and reentries by respondents than other data items. The higher rates for the first two 
questions - each over ten percent - may be due to respondents needing a few questions to orient 
themselves to the system. The payroll and hours questions generally have the greatest number 
of digits, so we suspect that data entry errors are more likely to occur, causing the question 
to be reread and the answer to be reentered. 

Figure 2 provides data for the first two questions on month and employment for repeated 
questions after no answer (repeat) and after lack of confirmation (reenter), separately. It was 
not possible to acquire data separately for the other data items. The CES touchtone system 
requires respondents to enter at least their report identification number, the month and employ- 
ment. The system will accept item nonresponse for the other data items. The mandatory entering 
of month and employment allowed the separation of repeated questions after no answer versus 
after a respondents lack of confirmation of the answer provided. Of respondents with problems 
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Month AE WW PW Payroll Hours 


BH 90% Contid. int. ff Point Estimate 


AE = All Employees 
WW = Women Workers 
PW = Production Workers 


(N = 1,203 per item) 
Figure 1. Machine Recorded Data - Percent of Questions Repeated/Reentered 


12 


10 


Month-Repeated Month-Reentered AE-Repeated AE-Reentered 


9 90% Confid. Int. 4 Point Estimate 


AE = All Employees 
Repeated = No Answer Within 2 Seconds 
Reentered = Correction of Answer 


(N = 1,203 per item) 


Figure 2. Machine Recorded Data - Percent of Questions Repeated Versus Reentered for Ist Two Data 
Items 
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on the month question, almost all were due to repeats, that is, two seconds passed without 
a response. On the other hand, problems with the employment question were almost evenly 
split between repeats after no previous answers and reentries after lack of confirmation of the 
previous answer. 

Only two percent of the calls received by TDE each month included just a report identifica- 
tion number. During these calls, respondents had simply hung up, or could have been cut off 
the system. 

In addition, the TDE system records all calls received that include at least the report iden- 
tification number, month and employment. Using the TDE component of the record-check 
data discussed earlier, we identified respondents with more than one call during a month and 
coded reasons for the call-backs. In all, about four percent of the respondents called the system 
more than once in a given month. Most of these respondents provided data items which were 
not supplied on the initial call (2%). An additional one percent provided corrections to some 
data items in addition to new data items. Many of these respondents appeared to have had 
problems with entering the data the first time. Another one percent of the respondents called 
back only to provide corrections to data items previously supplied or provided identical data. 
These calls were often several days later, possibly implying that new data had been obtained 
from their records. In the case of the identical data, respondents may have forgotten whether 
or not they had previously reported their data. The system currently accepts the data with the 
latest date and time, although analysts are provided a list of respondents with duplicate records 
for review, and if necessary, correction. 

A common reason for callback seemed to be related to the ‘‘enter 1’’ to confirm after each 
data item is entered. Many of the respondents who corrected their data had ‘‘1’’ in the data 
field prior to the callback, and some other response afterwards. Callbacks were twice as 
common with first-time respondents on the touchtone system than for respondents who were 
““experienced’’ users. 

A few respondents called in their data three times for a given month, and one respondent 
called in data four times. These respondents seemed to be having difficulties with the system, 
but finally reported all of their data correctly. 

Overall, these data suggest that respondents are having some difficulties with the system 
(more than they admit to during the respondent debriefing interview). Some steps could be 
taken to help alleviate some of the problems. These include providing more time for respondents 
to answer, providing better instructions, and trying to improve the confirmation of data entry 
method. In addition, being able to go back to a data item might solve some of the problems. 


5.3 Respondent Debriefing Survey 


BLS interviewers conducted a telephone debriefing survey with TDE respondents during 
1990. Given the human-factors literature discussed earlier, some of the questions focused on 
understanding and pace of the digitized voice. The results from the machine-recorded data 
showed a substantial number of repeats and retries of questions, thus, questions were devel- 
oped to address that topic. In addition, respondents were asked to rate the TDE system and 
answer questions relating to systems design. 

The results from the debriefing survey are presented in Table 4. Respondents expressed little 
difficulty in comprehending the digitized voice. About 97 percent said the voice was very 
understandable, and all respondents indicated that it was easy to understand the numbers as 
the voice read them back for confirmation. During the first two months of the survey, we asked 
respondents about the pace of the interview. Most of the respondents said the pace was about 
right (88%), although about ten percent felt it was too slow. Our suspicion throughout the 
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Table 4 
Results of Debriefing Survey* 


Voice understandable 97% 
Easy to understand #’s read back 100% 
Pace about right 88% 
Never reentered numbers 60% 
Never repeated questions 83% 
Never had poor telephone connection 93% 
Used speed enhancement feature 63% 
Instructions adequate 98% 
TDE experience very favorable 93% 


*N = 411, except for the pace and question repeat items. 
Approximately 177 respondents were asked about pace and 
209 were asked about the repeating of questions. 


study was that voice comprehension was much less a problem than were difficulties carrying 
out the task. While it was difficult to separate out the two in the record-check data, the 
debriefing interviews lend support to our suspicion. 

For task difficulties, 60 percent of respondents said they never had to reenter numbers, while 
most of the others indicated they had to reenter numbers sometimes. When asked the reasons 
for reentering numbers, a majority indicated they had accidently entered a wrong number. 
Others said they did not have enough time, were distracted, or entered their numbers too fast. 
In the latter several months of interviewing, respondents were asked about the repeating of 
questions (without reentering data). About 83 percent of the respondents said they never found 
it necessary to repeat questions. Of the 17 percent who repeated questions, the majority said 
they were distracted, while others said they did not have enough time. 

Most respondents had little difficulty with telecommunications failure, as 93 percent said 
they never experienced a poor telephone connection when using TDE. Of the respondents who 
did get a poor connection, most said it happened only once. A large number of the respondents, 
63 percent, used the pound sign, a feature of the system designed for speeding up the reporting 
of data. 

Nearly all respondents said the instructions sent to them as they began TDE were adequate. 
Overall, respondents seemed satisfied with the TDE system - approximately 93 percent rated 
their experience using TDE as very favorable. 


6. DISCUSSION 


The data show few serious problems with the TDE mode of data collection. Record-check 
data indicate some item nonresponse error, which is associated with first-time users. Entering 
additional or incorrect digits appears to be the most serious problem affecting the data items. 
However, in a panel survey, longitudinal edit checks could reduce this error, as could logical 
edit checks in all surveys. In addition, the rounding of data needs to be addressed in respondent 
instructions. Both the record-check and machine-recorded data show that there are more 
difficulties with longer strings of numbers, probably in both entering data and verifying 
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incorrect data. The latter could indicate difficulty remembering longer number chains during 
verification, as comprehension of numbers appeared to be good, i.e. respondents said they 
easily understood numbers being read back for confirmation. 

Record-check data show that establishments may have carried over their confirmation of 
the month into the all-employee question. In addition, the machine-recorded data indicate 
respondents often do not respond to the month question the first time it is asked. Since 
respondents appear to be using their survey forms as they enter data, it is likely that moving 
from the identification number at the top of the form, to the month and data items further 
down the form, they require extra time to locate themselves. This problem could be solved by 
placing all information that needs to be entered in one location on the survey form. This might 
reduce the number of question repeats for the ‘‘month’’ item and potentially lower costs by 
reducing the length of calls. Question repeats for other items might be reduced by giving 
respondents more time to respond, since they report they were distracted from the task. How- 
ever, since most respondents feel the pace of the system is about right, and many are using 
the speed enhancement feature, adding more time could cause frustration. Probably little can 
be done to reduce the number of reentries, as respondents indicate they have entered a wrong 
number and need to correct it. 

The data show that errors are reduced with experience. This indicates that a panel survey 
may be best for this mode of data collection. For surveys requiring numeric or yes/no responses, 
we believe touchtone also has great potential. The errors are not extremely serious, and 
respondents rate their experiences using TDE very favorably. TDE may be particularly attractive 
to business respondents, who can call at convenient times, rather than be interrupted by 
telephone calls requesting data. However, for some surveys, self initiation and the lack of human 
contact may be problems which would contribute to nonresponse error. 


Although respondent acceptance of touchtone collection is very favorable, there are some 
steps which can be taken to make the system better. These include: 


- giving respondents enough time to key enter their data, especially for the first few questions 
and those which have a long string of digits, 


- investigating ways to improve the confirmation of data items, and 


- providing longitudinal edit checks to detect reporting of dollars and cents and other gross 
errors. The edits could be built into the TDE system with appropriate questions/probes to 
respondents to correct or confirm their answers. 


BLS has used touchtone collection with one other survey. This survey was a small sample 
follow-up of business establishments who had participated in a Survey of Employer Drug 
Assistance Programs in 1988. The follow-up survey in 1990 was intended to determine if any 
substantial changes had occurred in the percentage of establishments providing employer drug 
assistance programs over the past two years. These establishments were mailed a short survey 
questionnaire requesting numeric or yes/no answers and encouraged to report their data by 
touchtone telephone. At the end of the first several weeks of the survey, approximately 20 
percent of the establishments had reported their data by touchtone, and an equal amount by 
mail. TDE was not used after nonresponse follow-up activities began - about two weeks after 
the initial mailout. The remaining data were collected by telephone (CATI). 

We believe that other surveys with time dependent data can take advantage of the time and 
keypunch savings of touchtone data collection. The mode may communicate the importance 
of timeliness to the respondent. This paper indicates that measurement errors are controllable 
using touchtone collection. 
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Given the timeliness and lower costs of touchtone data collection, we expect it will be used 
more extensively in the future. We know of two current projects testing touchtone recognition 
in a survey setting. Statistics Canada is testing a touchtone data collection system for the Survey 
of Employment, Payroll and Hours. In addition, a touchtone system for a survey of AT & T 
customers is being developed at Bell Laboratories (Wendler 1990). 

BLS is also experimenting with the use of voice recognition technology for data collection 
in the CES survey (see Winter and Clayton 1990). While touchtone telephones are increasingly 
available, we estimate that between 10 to 20 percent of our respondents do not have touchtone 
telephones. Once speaker-independent voice recognition technology reaches an acceptable level 
for the ten digits needed to report CES data, we expect users will prefer it over touchtone 
collection. Further work on measurement errors associated with voice recognition technology 
needs to be undertaken. 
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Avoiding Sequential Sampling with Random Digit Dialing 


J. MICHAEL BRICK and JOSEPH WAKSBERG! 


ABSTRACT 


The Mitofsky-Waksberg procedure is an efficient method for selecting a self-weighting, random digit 
dialing (RDD) sample of households. The Mitofsky-Waksberg procedure is sequential, requiring a constant 
number of households be selected from each cluster. In this article, a modified Mitofsky-Waksberg 
procedure which is not self-weighting or sequential is described. The bias and variance for estimates derived 
from the modified procedure are investigated. Suggestions on circumstances which might favor the 
modified procedure over the standard Mitofsky-Waksberg procedure are provided. 


KEY WORDS: Random digit dialing; Telephone sampling; Cluster sampling; Trimming. 


1. INTRODUCTION 


The Mitofsky-Waksberg procedure for selecting random digit dialing samples of households 
(Waksberg 1978) is frequently used for sample selection in telephone surveys. As described 
in the Waksberg paper, it is an efficient method of producing a self-weighting sample, that 
is, One in which all telephone households have the same probability of selection (except for 
households with more than one telephone number). The efficiency is due to the sharp reduc- 
tion in the proportion of nonhousehold telephone numbers that have to be dialed in order to 
identify sample households. 

The Mitofsky-Waksberg procedure is a two-stage sample design. In the first stage, a sample 
of clusters is chosen where the clusters consist of blocks of 100 telephone numbers, or multiples 
of such blocks. The clusters (or blocks of 100 telephone numbers) are first selected with equal 
probability. One telephone number is chosen at random in each cluster and dialed. If the number 
is that of a household, the cluster is retained. Otherwise, it is rejected. The second stage is the 
selection of households within the retained sample clusters. For the self-weighting feature of 
the sample to apply, a constant number of households per cluster is required. Some organiza- 
tions (including Westat Inc.) generally go a little further and specify a constant number of 
interviewed households per cluster (or screened households if the first part of data collection 
is screening). The rationale is that substituting another randomly selected household within 
the same cluster for each nonrespondent is a reasonable way of reducing nonresponse bias. 

There is an awkward operational feature to this system. It sometimes takes a fairly large 
number of callbacks to determine whether or not a telephone number is residential, particularly 
for numbers that repeatedly ring with no answer. Even more are needed to learn which 
households cooperate. Such determinations must be made for an initially selected sample to 
ascertain which clusters require more telephone numbers to achieve the desired cluster size and 
how many telephone numbers have to be added. In effect, a sequential scheme is necessary 
for each cluster, where all previous cases need to be cleared up before it is known whether the 
sample needs to be increased. This process is particularly inconvenient when there is a tight 
time schedule for data collection. 

Several attempts to modify the Mitofsky-Waksberg method have been proposed which reduce 
or eliminate the sequential features of the plan. Potthoff (1987) developed a generalization 
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of the Mitofsky-Waksberg technique in which c telephone numbers are chosen per cluster in 
determining whether to retain the cluster, whereas Mitofsky-Waksberg use only one. A self- 
weighting sample is achieved by having clusters in which only one of the c telephone numbers 
dialed continue with a sampling plan that includes having a fixed number of households per 
cluster, and the remaining clusters having a fixed number of telephone numbers. The latter 
group of clusters does not require a sequential approach. Potthoff reports that in practice most 
clusters will fall into the second class so that the sequential operations, although not eliminated, 
are sharply reduced. 

Lepkowski and Groves (1986) describe a sampling method in which blocks of telephone 
numbers with more than a trivial number of telephone numbers listed in directories (and other 
sources, if available) are selected with probability proportionate to the number of listed 
numbers. Blocks of numbers which contain zero or very few telephone numbers are sampled 
through the Mitofsky-Waksberg procedure. Sudman (1973) had previously proposed sampling 
blocks of numbers with probability proportionate to the numbers listed in directories, but 
without making any provision for empty blocks (which could have unlisted numbers). Drew 
and Jaworski (1986) describe an RDD survey carried out in Canada in which purchased counts 
of residential numbers (both published in directories and nonpublished) were used as measures 
of size. Since the counts were considered as virtually complete, there was no need to sample 
empty blocks. As far as we are aware, there is no way of getting virtually complete counts of 
residential numbers in the U.S. 

Neither the Potthoff nor the Lepkowski-Groves sample design completely eliminates the 
need for a sequential process, although both appear to reduce the portion of the sample which 
requires it. There are also some other disadvantages to the two procedures. The Potthoff 
technique appears to be rather complex — as far as we know it has not been used much for RDD 
surveys. For national surveys, the Lepkowski-Groves technique involves the purchase of a direc- 
tory list covering the total U.S. and processing it to obtain measures of size. Such commercial 
lists are available, but they are expensive. Furthermore, a number of recent reports indicate 
the percentage of all residential numbers that are listed in directories is not very high, and is 
rapidly decreasing. Tucker (1989) describes an analysis of listed numbers in a group of U.S. 
counties and cities which shows listing rates varying from 48 to 62 percent. An article by Linda 
Piekarski (1990) states that if the rate of increase of unlisted numbers continues at the current 
level, ‘‘as many as 62% of the nation’s households may be unlisted by the year 2000.’’ The 
measures of size thus are probably only moderately correlated with the actual number of 
households in a working block. 

Waksberg has suggested an alternative modification of the Mitofsky-Waksberg procedure 
(Waksberg 1984) which completely eliminates the need for sequential sampling. Westat has 
used this method in a large number of studies using RDD. Cummings (1979) had previously 
used the same procedures as a result of an error in implementing the Mitofsky-Waksberg 
procedure. Cummings did not recognize its usefulness in avoiding sequential sampling and did 
not explore its features for use in other surveys. We describe the method and its mathematical 
and statistical properties. 


2. ALTERNATIVE METHOD OF ESTABLISHING CLUSTER SIZES 
WITH MITOFSKY-WAKSBERG TECHNIQUE 


As indicated earlier, the Mitofsky-Waksberg technique requires a constant number of sample 
residential numbers per cluster (or block of numbers) to produce a self-weighting sample. The 
alternative that is proposed is to use a constant number of telephone numbers per cluster for 
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the sample (K). The first stage of selection is unchanged. (The first-stage selects clusters with 
probability proportionate to the number of households.) With a constant number of telephone 
numbers per cluster, the sample numbers can be designated in advance eliminating the sequential 
process. We note that followup effort is still necessary to determine which sample telephone 
numbers are residential, both in the first and in the second stages of sampling. However, this 
has to be done for a fixed set of telephone numbers. A sequential process is not involved. 


The alternative procedure does not produce a self-weighting sample. Since the first stage 
is selected with PPS, the probability of a cluster being selected is r N;/100 where r is the 
sampling rate for selection of the clusters, that is, the first stage selection rate, and N; is the 
number of residential numbers in the ith cluster. The weight should be proportional to N;"', 
but since N; is not known, it is taken to be proportional to n;_', the number of sample 
households in the cluster. 


This modification of the Mitofsky-Waksberg method has good features for survey opera- 
tions. It is simple. The sample can be virtually preselected and no costly control operations 
are needed. Although weighting is required, the weights are directly available from the sample 
data, and they can be mechanically produced without any extensive professional oversight. 


There are, however, some serious problems. First, there is a bias when N;_! is estimated 
by K/100n; where K is the number of telephone numbers selected per cluster (a constant 
number in all clusters). The bias is fairly small, but it does exist. It cannot be eliminated or 
reduced by minor modifications of the weights, such as using 1/(n; + f) instead of n;_!, with 
**t’? denoting a fixed constant. Secondly, the introduction of variable weights increases the 
variances of the estimates substantially. (The increase is not so much caused by the weights 
as by the fact they reflect variable probabilities of selection.) Finally, the modification loses 
one of the useful features of the Mitofsky-Waksberg method - the ability to fix the exact sample 
size desired. The Mitofsky-Waksberg method’s use of a constant number of households per 
cluster means that any desired sample size can be obtained by selecting a sample with the 
appropriate number of clusters. With the modification, the sample size becomes a random 
variable, which generally will not be exactly equal to the desired sample size. Although the devia- 
tions are usually small, the ability to achieve exact target sizes is useful when contracts or budget 
commitments require the survey organization to satisfy exact target requirements. We discuss 
these issues in Sections 3 and 4. 


Before going on to a discussion of the variances and biases, it is useful to examine the dis- 
tribution of cluster sizes in the U.S. Tables 1 to 3 show estimates of such distributions prepared 
from data reported in two large national U.S. surveys conducted via RDD by Westat Inc. Both 
of these surveys used the modification of the Mitofsky-Waksberg procedure described above. 
The sample for the survey summarized in Table 1 was selected in 1986 and consisted of 2,427 
clusters (retained after first-stage sampling) with 15 telephone numbers per cluster, or 36,405 
total numbers. There were 18,756 completed screeners, 2,396 refusals, 1,727 nonresponse for 
other reasons, and 13,526 nonresidential or nonworking numbers, ring no answers, and cases 
that could not be classified. The analysis is restricted to the 18,756 completed cases. The data 
in Tables 2 and 3 are based on a 1989 sample of 1,000 clusters with 30 telephone numbers per 
cluster or 30,000 telephone numbers, of which 19,586 were residential with screeners completed. 
Table 2 shows the distribution of the 15,030 completed cases and Table 3 shows the distribu- 
tion of the 19,586 residential numbers found in the 1,000 clusters. The cluster weights shown 
are expressed as fi/n; where 7 is the average number of households per cluster. It seems useful 
to express them in this form since they then show the deviations from a self-weighting sample. 
The design effects only account for the increased variances arising from variable sampling 
fractions. They do not include effects of other aspects of the sample design. 
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Table 1 


Number of Completed Screeners per Cluster in 1986 Survey 
(Based on sample of 2,427 clusters with 15 telephone numbers per cluster) 


Number of Average Household Distribution Cluster Distribution 
Completes Cluster . : 
per Ciustcs Weight! Frequency Percent HS Frequency Percent aot 
0 XX 0 0 0.0 62 2.6 2.6 
1 73937 54 0 0.3 54 2:2 4.8 
2 3.977 106 0.6 0.9 53 Fz 7.0 
3 2.64 258 1.4 Phe 86 ee 10.5 
4 1.98 440 233 4.6 110 4.5 15.0 
5 1.59 810 4.3 8.9 162 6.7 247 
6 1:32 1,290 6.9 15.8 21S 8.9 30.6 
7 e138 1,960 10.5 26.2 280 KS 42.1 
8 0.99 2,656 14.2 40.4 332 Ey 55.8 
9 0.88 2,862 15.3 55.6 318 131 68.9 
10 0.79 2,990 15.9 71.6 299 | bea | 81.2 
11 0.72 peg A 4 14.5 86.1 247 10.2 91.4 
12 0.66 1,548 8.3 94.3 129 5:3 96.7 
13 0.61 780 4.2 98.5 60 235 99.2 
14 0.57 210 1 99.6 | ts 0.6 99.8 
15 0.53 15 0 100.0 5 0.2 100.0 
Total 18,756 100.0 XX 2,427 100.0 XX 
Mean cluster size? 7.93 
Design effect* 11 


! The cluster weight is the mean cluster size (i.e., 7.93) divided by the number of completes in the /-th cluster. 
Trimming the weights would bring these weights down to 3. 
The mean cluster size is the average over the 2,365 clusters with one or more completed screeners. 
The design effect is reduced to 1.12 if the maximum weight is 3. 


It should be noted that Table 1 is based on a sample of 15 telephone numbers per cluster 
and Tables 2 and 3 used 30 telephone numbers per cluster. Estimates of the percent residential 
in a cluster based on 15 telephone numbers will, of course, be subject to a higher sampling 
error than an estimate based on 30 telephone numbers. However, the number of clusters used 
in Table 1 was more than twice those in Tables 2 and 3 which should largely offset the effect 
of the different cluster sizes. 

There are two differences between Tables 2 and 3. One is that Table 2 shows the distribu- 
tion of completed screeners (as does Table 1) while Table 3 is based on all sample households. 
The use of only completed cases in Table 2 reduces the estimate of the average number of 
households per cluster and shifts the entire distribution. In addition, it introduces more 
variability to the estimates of the distribution shown because the distributions reflect sampling 
errors of both the distribution of households per cluster and the distribution of nonresponse 
rates per cluster. The second difference is that Table 2 (and Table 1) is expressed in terms of 
the number of cases per cluster and Table 3 shows the distributions by the percentage of residen- 
tial numbers per cluster. It was convenient to express Table 3 in that form for analyses described 
later in this report. 
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One other feature of the percentages shown in Tables 1 to 3 should be noted. They reflect 
the size distributions of clusters which fell into the sample, not the distribution of clusters in 
the U.S. The use of probability proportionate to size sampling results in an oversampling of 
clusters with a high proportion of residential numbers and an underrepresentation of clusters 
with a small number. It is possible to convert the distribution from one that represents the 
sample to one that represents the population by multiplying each percentage by the cluster 
weights and computing the percentage distribution of the resulting figures. Since the weights 
are exactly proportional to the reciprocal of the number of completes per cluster, it turns out 
that converting the household distribution so that it represents the distribution in the popula- 
tion produces the percentages shown in the cluster distribution. The cluster distribution in the 
sample is thus the same as the household distribution in the population. 

We show distributions of both all-sample households and completed cases because both 
are of interest to researchers. The Table 3 data have been used for the analyses in Sections 
3 and 4. 


Table 2 


Number of Completed Screeners per Cluster in 1989 Survey 
(Based on sample of 1,000 clusters with 30 telephone numbers per cluster) 
Number of Average Household Distribution Cluster Distribution 


Completes Cluster Cumulative Cumulative 


per Cluster Weight! Frequency Percent Bereent Frequency Percent Percent 

0 XX 0 0 0.0 8 0.8 0.8 

lor 2 7.514 6 0 0.0 3 0.3 1.1 

3 or 4 4.33? 37 0.2 0.3 10 1.0 2.1 

5 or 6 Pf be) 126 0.8 jee 22 Dee 4.3 

7 or 8 2.02 403 Ze 3.8 53 533 9.6 

9 or 10 159 688 4.6 8.4 72 TZ 16.8 

11 or 12 132 13325 8.8 17,2 115 115 28.3 

13 or 14 hed 2 1,987 132 30.4 147 14.7 43.0 

15 or 16 0.98 2,636 | 50.0 170 17.0 60.0 

17 or 18 0.85 2,692 17-9 65.9 154 15.4 75.4 

19 or 20 0.78 2,387 15-9 81.8 123 12.3 87.7 

ZL or 22 0.70 1,673 11.1 92.9 78 7.8 pre 

23 or 24 0.64 816 5.4 98.3 35 35 99.0 

25 or 26 0.55 254 i / 100.0 10 1.0 100.0 

27 or 28 XX 0 0 100.0 0 0 100.0 

29 or 30 XX 0 0 100.0 0 0 100.0 

Total XX 15,030 XX XX 1,000 XX XX 
Mean cluster size? fe) 
Design effect* 1.33 


! The cluster weight is the mean cluster size (i.e., 15.15) divided by the number of completes in the i-th cluster. 
Trimming the weights would bring these weights down to 3. 
The mean cluster size is the average over the 992 clusters with one or more completed screeners. 
The design effect is reduced to 1.12 if the maximum weight is 3. 
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Table 3 


Proportion of Residential Numbers per Cluster in 1989 Survey 
(Based on sample of 1,000 clusters with 30 telephone numbers per cluster) 


Proportion of | Average Household Distribution Cluster Distribution 


Residential Cluster 


: Cumulative Cumulative 
nos. per Cluster Weight! Frequency Percent 


Renicnt Frequency Percent Bareert 


0 Xx 0 0.0 0.0 5 0.5 0.5 

.001 to .049 21.762 5 0.0 0.0 3 0.3 0.8 
.05 to .099 8.702 18 0.1 0.1 6 0.6 1.4 
.10 to .149 5.222 4] 0.2 0.3 9 0.9 2.3 
.15 to .199 3.732 48 0.2 0.6 8 0.8 3.1 
.20 to .249 2.90 53 0.3 0.8 7 0.7 3.8 
.25 to .299 2.37 144 0.7 1.6 16 1.6 5.4 
.30 to .349 2.01 178 0.9 es 17 7 7.1 
35 to .399 1.74 408 pa 4.6 34 3.4 10.5 
.40 to .449 1.54 459 3 6.9 34 3.4 13.9 
.45 to .499 1.37 840 4.3 11.2 56 5.6 19.5 
.50 to .549 1.24 1,040 5.3 16.5 63 6.3 25.8 
.55 to .599 1.14 1,926 9.8 26.3 107 10.7 36.5 
.60 to .649 1.04 2,126 10.9 37.2 109 10.9 47.4 
.65 to .699 0.97 3,255 16.6 53.8 155 15.5 62.9 
.70 to .749 0.90 2,610 13.3 67.1 116 11.6 74.5 
.75 to .799 0.84 3,022 15.4 82.6 126 12.6 SFei 
.80 to .849 0.79 1,556 7.9 90.5 61 6.1 93.2 
.85 to .899 0.75 1,458 7.4 98.0 54 5.4 98.6 
.90 to .949 0.71 399 2.0 100.0 14 1.4 100.0 
.95 to .999 XX 0 0.0 100.0 0 0.0 100.0 
Total XX 19,586 100.0 XX 1000 100.0 Os 

Mean cluster size? 19.68 
Design effect* 1.28 


! The cluster weight is the mean proportion in a cluster (i.e., 0.653) divided by the proportion of residential numbers 
in the i-th cluster. 
Trimming the weights would bring these weights down to 3. 
The mean cluster size is the average over the 995 clusters with one or more residential numbers. 
The design effect is reduced to 1.12 if the maximum weight is 3. 


3. VARIANCE IMPLICATIONS OF THE MODIFIED 
MITOFSKY-WAKSBERG METHOD 


In the standard Mitofsky-Waksberg method the variance of a sample estimate is dependent 
upon the number of households selected per cluster and the homogeneity of the households 
within and between clusters. The variance for a cluster sample can be written as the variance 
for asimple random sample multiplied by [1 + o(# — 1)], where pis intraclass correlation 
and A is the average number of households per cluster. Since telephone clusters are often related 
to geographic areas and tend to be somewhat homogeneous, selecting a large number of 
households per cluster can be inefficient. 

When the modified Mitofsky-Waksberg method is used, another source of variance is 
introduced because the number of households selected per cluster is allowed to vary from cluster 
to cluster. As pointed out in Section 2, the denominator of the second stage probability of selec- 
tion does not cancel with the number of households in the cluster (which is proportional to 
the probabilities in the first stage) and the overall probabilities of selecting households vary 
from cluster to cluster. 
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The variability among clusters in the overall household sampling rates causes the variances 
of the estimates to be larger than those in the standard Mitofsky-Waksberg method where 
each household has the same probability of selection. Methods for estimating the increase in 
the variance of an estimate arising from unequal probabilities of selection are discussed by 
Kish (1965) and by Waksberg (1973). A simple approximation to the variance of an estimate 
under an unequal weighting scheme (where the weights do not reflect variable sampling rates 
in strata deliberately chosen to reduce sampling variances) is the sampling variance which would 
occur with a self-weighting sample multiplied by a variance inflation factor (VIF), given by 
VIF = {1 + Relvar(weights)}. We will use this approximation below to investigate the 
variance implications associated with the modified Mitofsky-Waksberg method. 

The relative variance of the weights was computed by partitioning the process into two 
components. First, the mean and variance of the weights were computed conditioned upon 
sampling from a truncated (since zero households cannot be obtained if the cluster is sampled 
in the first stage) hypergeometric distribution, defined by the household density in the cluster 
and the cluster sample size. The unconditional mean and variance of the weights were then 
computed by integrating over the distribution of households in the sampled clusters shown 
in Table 3. The distribution of households in the sample is critical in the evaluation of the VIF. 

The natural weight assigned to a household in the modified Mitofsky-Waksberg is propor- 
tional to n;_', where 7; is the number of households observed in sample cluster i. This weight 
can vary by factors which range from as little as 1/K to 1, where K is the number of telephone 
numbers selected in a cluster. The average weight is roughly 1.5/K, since about 65 percent of 
numbers in the sampled clusters are residential. 

If the number of telephone numbers sampled per cluster is between 5 and 30, then the increase 
in variance due to the weighting is about 30 percent. The VIF decreases slightly as the number 
sampled per cluster increases beyond 30, reaching approximately 17 percent when all the 
numbers in the cluster are sampled. 

The VIF or the relative variance of the weights is a function of the distribution of the number 
of households across clusters and random sampling variability within the clusters. This decom- 
position is made explicit by expressing the variance of the weights as the sum of the mean of 
the conditional variance of the weights and the variance of the conditional mean of the weights, 
where the conditioning is with respect to the household density of the cluster. 

When the cluster sample size is small, the mean of the conditional variance is the dominant 
component of the overall variance. As the cluster sample size increases, the variance of the 
conditional mean becomes more dominant. This is why the relative variance of the weights, 
shown in the first row of Table 4, is not a monotonic function of the cluster sample size. 


Table 4 


Approximate Variance Inflation Factors (VIF) for Modified 
Mitofsky-Waksberg Random Digit Dialing Samples 


Cluster Sample Size (K) 


Weight ee ere a er eer ieee iki ace 

5 10 30 60 100 
1/n; 134 1.34 ee 1.23 ky 
ACh; + .5) 1.18 124 1.20 1.18 1.16 
tiie 1) 1.12 n15 1.16 115 1.14 


WA CS am) 1.07 1.09 1.11 We | 
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Variances Using Different Weights 


Weights other than ones proportional to the inverse of the number of households were also 
examined to determine their impact on the bias and variance of the estimates. Many of the 
alternative weights studied were derived from variance stabilizing transformations suggested 
for binomial variables. 

Of all the alternatives examined, the estimators with the best bias and variance properties 
involved simple adjustments of the natural weight. In particular, adding a small constant 
to the observed number of households (estimators of the form (n; + t)~! where fis .5, 1, 
or 2) resulted in reducing the increases in variance due to differential weighting. The addition 
of the constant reduces the range of the weights by cutting the values of the largest weights 
while only slightly modifying the weights for clusters where more households are found in the 
sample. 

Table 4 shows the VIF for the estimators of the form (n; + ¢)~! for different numbers 
of telephone numbers sampled per cluster. The table also is based on the household and cluster 
distributions shown in Table 3. It is clear from the table that a substantial reduction in the 
variance due to unequal weighting can be achieved by using (n; + 1)~', rather than the 
natural estimator. This is especially true for RDD designs which sample 30 or fewer telephone 
numbers per cluster. The increase in variance due to differential weighting for (n; + 1) ~'is 
only 16 percent when 30 numbers are selected per cluster as opposed to a 29 percent increase 
when the natural estimator is used. 


Variances with Trimmed Weights 


A practice that is often used to mitigate the variance inflation associated with varying weights 
is the truncation of very large weights. This truncation, or trimming of weights, is usually fixed 
at a weight above which relatively few observations are found. In many Westat RDD samples, 
weights that exceed two or three times the mean weight have been truncated. For this research, 
we have examined weights truncated at about 3 times the mean weight. For samples of 10 per 
cluster, the weights were truncated at 2 times the mean weight because so few observations 
are affected otherwise. 

Table 5 shows the VIF for the estimators for different cluster sample sizes when the weights 
are trimmed at three times the mean weight for n;'!. The VIF’s for samples of 5 per cluster 
are not given because the truncation point in samples of this size is nearly at unity, the largest 
possible weight. 


Table 5 


Approximate Variance Inflation Factors (VIF) for Modified Mitofsky-Waksberg 
Random Digit Dial Samples with Trimmed* Weights 


Cluster Sample Size (K) 


Weight SE orEnEETEEEEEEEEEEEEE REE EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE 

10 30 60 100 
1/n; 1.12 tefl 1.09 1.09 
1/(n; + .5) 1.11 1.10 1.09 1.09 
Wun 1h 1.09 1.10 1.09 1.09 
1/(n; + 2) 1.07 1.09 1.09 1.08 


* All weights trimmed at 3 times the mean weight, except samples of 10 trimmed at 2 times the mean. 
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The tabled values show trimming substantially reduces adverse impact of the differential 
weights on the variance of the estimates. The most dramatic reduction is for the natural 
estimator; its VIF is reduced by over 50 percent by the use of trimming. The VIFs for the other 
estimators are improved somewhat, but the reductions are less striking since they already had 
smaller VIF’s than the natural estimator. Trimming has the potential of introducing biases 
which may counteract the advantage in variance reduction. Biases are discussed in Section 4. 


Variances with Augmented Sampling 


A third approach to reducing the variability of the weights is the use of augmented sampling. 
Large weights occur when the number of households identified in the cluster is small relative 
to the expected number of households per cluster. To reduce the chance for this happening, 
an augmented sampling procedure can be used. If the number of households in a cluster is 
smaller than a fixed number (say less than one third of the mean number per cluster), then 
the sample size in the cluster can be doubled or increased by some other amount. 

This procedure could be iterated to insure that the number of households per cluster reaches 
a specified limit or until all numbers in the cluster are used. The obvious disadvantage of this 
iterative plan is that it requires monitoring sample yield by cluster and the very fact that it is 
sequential. Another disadvantage of the method is that it results in sampling more telephone 
numbers from clusters that have a lower household density (the ones most likely to need 
augmentation), hence reducing productivity. 

Despite the operational shortcomings of the augmented sampling approach, we did a limited 
examination of the method. Since the results for the augmented sample approach was not better 
than trimming the weights, this method is not discussed further. 


4. BIAS IMPLICATIONS OF THE MODIFIED 
MITOFSKY-WAKSBERG METHOD 


The increase in variance is just one of the consequences of using the modified Mitofsky- 
Waksberg method of sampling. Another important feature of the method is the bias in the 
resulting estimates. If a fixed sample size is selected in a cluster and no weight adjustment is made, 
the variance of the estimates are not increased but the bias has the potential of being very large. 

The unbiased weight (W,,) for the modified method is 


100 100 
— Ga 


Y= 
rN; K 


’ 


where the terms are as defined above. The problem is that N; is unknown and does not cancel 
with the second stage term, as it does in the standard Mitofsky-Waksberg method. Weights 
are therefore introduced in an effort to reduce the bias. 

We refer to the estimator which uses a weight of n;_! as the natural estimator because n;/K 
is an unbiased estimator of N;/100 in sampling from a binomial or hypergeometric distribu- 
tion. (We use the weight of n;_' although the weight is actually K/100n;. Since K/100 is a con- 
stant, the relationship among the weights are not affected by using n;_'.) This weight appears 
to be the natural estimator despite the fact that n;-' is not unbiased for N; ' unless all 100 
numbers are selected in a cluster. The bias of nee is discussed in literature; for example, see 
the discussion on stratification after sampling in Hansen, Hurwitz and Madow (1953). No 
simple unbiased estimator, certainly none of the form (n; + ¢)~}, is likely to exist for all 
possible cluster sample sizes. 
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Figure 1. Mean Weights of Estimators Conditional on the Proportion Residential with Shaded Histogram 
of Proportion of Households in Cluster 
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One of the ways to examine the potential bias is by comparing the expected value of the 
estimators (the mean weight using estimators of the form (n; + t)~!) with the unbiased 
weight, W,,. Since both the unbiased weight and the expected value of the estimators are 
functions of N;, we will begin by investigating these quantities conditioned on Nj. 

Figure | shows the graph of the unbiased weight and the mean weights, using the estimators 
n; ' and (n; + 1)~!, when there are K = 10 telephone numbers selected per cluster. The 
constant cluster sampling rate, 7, has been omitted from all of the weights. A logarithmic scale 
has been used for the mean weights because of the range in W,,. 

The graph clearly shows that the biggest differences between W,, and the mean weights for 
the two estimators are found when N,/100 is small. Once the residential density exceeds 20 
percent when (n; + 1) ~! is used, and 10 percent for n;—!, the differences are relatively minor. 
The graph shows that the weight (n; + 1) ~! is always smaller than W,, but this will not be 
true if poststratification is used. Poststratified weights are not used in the graph because 
poststratification really operates on the unconditional weights rather than the conditional 
weights shown here. The unconditional bias is addressed below. 

The shaded histogram in the figure shows the distribution of households from Table 3. It 
has been overlaid to illustrate the fact that the large differences in weights occur in clusters 
which account for a very small fraction of the sampled households. 


Bias in Sample Size and Bias in Estimates 


In nearly all RDD surveys, including those using the Mitofsky-Waksberg sample design, 
poststratification of the sample to known totals of persons or households is used. One of 
the prime reasons for using poststratification is to adjust the estimates to the levels associated 
with all persons, not just those in households with telephones. Massey and Botman discuss 
this and other benefits of poststratification in RDD surveys in Chapter 9 of Groves et al. 
(1988). 

Regardless of the reasons for using it, poststratification results in estimates that are equal 
to known totals irrespective of the weights applied to the individual households. Since this 
bias, which can be considered as bias in sample size, is always zero, it is difficult to find a 
single statistic that measures unconditional bias directly. To attack this problem, we will 
examine the relative contribution to the bias in sample size over the range of household density 
values. 

The following steps were taken to compute a measure of this contribution to bias in sample 
size. First, the different weighting functions or estimators were computed using the empirical 
household density shown in Table 3. Then, the estimates were poststratified to equal unity and 
the contribution to the total was computed for different values of N;/100. Finally, the relative 
bias in sample size was defined as the difference between the contribution to the total from 
the particular estimator and the contribution from the total using W,, as the weight. 

This measure thus takes into account both the difference in the weights for fixed values of 
N; and the distribution of households across all the values of N;. Thus, sampled households 
from clusters with values of N; that are rare will not contribute heavily to the relative bias in 
sample size even if they are associated with large differences in weights. 

To illustrate these computations, Figure 2 shows the relative bias in sample size for some 
estimators for samples of 30 numbers per cluster. One of the estimators uses the unadjusted 
weight, i.e., the weight is a constant for all households regardless of the number of households 
identified in a cluster. The relative bias in sample size for the estimator with unadjusted weights 
is much larger than when other weights are used. The unadjusted weight has relative biases 
in sample size that range from about —2 percent to +3 percent. 
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Figure 2. Relative Bias from Sample Size for Samples of 30 Cluster 


The size of the bias in the estimate of a characteristic is bounded by the size of the bias in 
sample size. In other words, the relative bias in the estimate can be no larger than the relative 
bias in the sample size. For almost all characteristics, this upper bound will not be attained. 
The upper bound is only attained when the characteristic and the residential density are perfectly 
correlated. Very high correlations are not likely in national samples, but might be more feasible 
in samples in restricted geographic areas. 

It can be seen that there are patterns in the biases; for example, the unadjusted estimator 
is uniformly too low in low proportion residential clusters and too high in clusters with a high 
proportion of households. When there are differences in the characteristics between low and 
high density clusters, the biases can be quite serious. The bias in estimates resulting from using 
unadjusted weights can be seen for some characteristics in Table 1 in Cummings (1979). The 
biases are not very large, but appropriate weighting will effectively eliminate them. 
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In general, the relationship between the estimate and the number of households in a cluster 
will be unknown and inconsistent across all the characteristics to be estimated. Therefore, a 
reasonable practice is to choose an estimator that has a relative bias in sample size that is small 
across the range of values of N;. If the relative bias in sample size for a set of the estimators 
is small, then the choice of estimators can be dictated by variance considerations. 


Biases Using Different Weights 


The relative bias in sample size were computed using different estimators for samples of 
5, 10, 30, and 60 telephone numbers per cluster. The relative bias in sample size is negligible 
for the cluster sample sizes of 30 and 60 numbers, except when the unadjusted weights are used. 
Any of the adjusted estimators could be used for cluster samples sizes of this size without incur- 
ring biases in the estimates. When 10 numbers are selected per cluster, all of the weights except 
the unadjusted one again perform reasonably well. The bias performance of (n; + .5)~! is 
especially encouraging. 

For the smallest cluster size studied, 5 numbers per cluster, the potential for bias is some- 
what greater. The natural weight, n;_!, has a somewhat lower bias in sample size than the 
weight (n; + .5)~!, but the difference is not very large. The relative bias in sample size for 
both of these weights is always less than one percent. For residential densities between about 
45 percent and 80 percent the bias is positive and elsewhere it is negative. This pattern might 
be problematic only for the few characteristics that are very highly correlated with residential 
density. 


Biases with Trimmed Weights 


The introduction of trimming can produce significant biases, depending on the relationship 
between the characteristics being estimated and the weights which are being trimmed. In some 
applications, the bias associated with trimming may limit the amount of trimming that can 
be applied and, hence, its usefulness for variance reduction. 

The relative bias in sample size was also computed for cluster samples of 10, 30 and 60 
numbers and the weights trimmed at about 3 times the mean weight. The trimming for samples 
of 10 numbers per cluster was done at a factor of 2 rather than 3 as described previously. 

The difference between the relative bias in sample size for the trimmed and untrimmed 
weights is largely inconsequential for all cluster sample sizes and most values of N;/100. The 
only noticeable difference occurred when the residential density is under about 10 to 15 percent. 
There is a slightly greater potential for bias in these regions. However, the relative bias in sample 
size for the trimmed weights is still much less than one percent at all residential density values. 


5. CONCLUSION 


The standard Mitofsky-Waksberg method is an effective method of producing a self- 
weighting, RDD sample of fixed size. However, the sequential monitoring of the number of 
cases per cluster is an awkward operational feature of this method. One of the consequences of 
the sequential monitoring of caseloads by cluster is that it is difficult to complete data collection 
in a tight time frame. The data collection period has to be flexible enough to allow for obtaining 
the appropriate number of cases in each cluster. The more extensive data collection period and 
the monitoring of caseloads also result in increasing costs. Another problem with the sequential 
operations is that the requirement for frequent monitoring of the caseloads can lead to frustra- 
tion arising from complications of combining sample selection and data collection operations. 
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The modified Mitofsky-Waksberg approach eliminates the sequential nature of the design 
and, with it, the need to monitor the work by cluster. A fixed number of telephone numbers 
are assigned to each sampled cluster in the modified method. Therefore, the costs associated 
with monitoring caseloads and a longer data collection period are not incurred. However, the 
modified Mitofsky-Waksberg method does introduce new components of bias and variance 
into the estimates. These statistical concerns should be addressed before the modified approach 
is used. 

Specific recommendations on when the standard or modified Mitofsky-Waksberg method 
should be used are difficult to formulate since they depend upon circumstances which vary 
from survey to survey. Guidelines for choosing between the methods are suggested below. 

A simple rule is that for surveys which require either very tight controls on sample size or 
a nearly self-weighting sample, then the standard Mitofsky-Waksberg approach is advisable. 
Even though the sample size in the modified method can be estimated relatively precisely, some 
variation, especially because of uncertainty of the nonresponse rates, can be expected. A self- 
weighting sample, which is not achieved when the modified Mitofsky-Waksberg method is used, 
also has some advantages in simplifying standard statistical analysis. 

Since the costs for standard and modified methods are different, it would be very useful 
to have cost-variance models to help evaluate the two methods. Unfortunately, the differences 
in costs of the standard and modified methods are not easy to quantify. In fact, the lack of 
reasonable cost models is a major and pervasive problem that limits the ability to establish 
optimal survey design. 

Because of lack of reasonable cost-variance models, we suggest some conditions in which 
one approach might be favored over another. One of the conditions that favors the modified 
approach is a relatively brief interview length. As the interview becomes longer, the cost savings 
associated with the modified method is likely to become smaller relative to the increases in 
variances of the estimates. 

The length of the interview is particularly important for surveys which screen households 
to find those with particular characteristics. For example, some RDD surveys screen households 
and only interview if a member is in a particular target group. In these situations, the screening 
interview is often very brief. The modified Mitofsky-Waksberg approach may be very beneficial. 
Surveys in which households are screened also tend to have large cluster sample sizes, and this 
improves the performance of the modified procedure. When 10 or more numbers are selected 
per cluster (equivalent to about 6 households per cluster), the biases in the estimates under the 
modified Mitofsky-Waksberg approach are virtually inconsequential and the increases in 
variance with trimming are only about 10 percent. Samples of 10 or more numbers per cluster 
are frequently acceptable for screening purpose although such large cluster sizes are typically 
inefficient for the interview sample, even when the intraclass correlation is small. 

Based on these factors, a general guideline is that the modified Mitofsky-Waksberg method 
can be recommended when households within the clusters must be screened. More specifically, 
the modified method with trimmed weights should be considered if the following conditions 
exist: (1) Ten or more numbers are sampled per cluster; and (2) the total cost for the modified 
method is at least 10 percent less than the standard method, or the data collection period is 
relatively short. If both of these conditions are not met, then the choice between methods must 
be made on evaluations of other survey requirements. 

When the cluster sample size is less than 10, the bias and variance arising from the use of 
the modified Mitofsky-Waksberg method are more serious concerns. Any characteristics 
correlated with the proportion of residential numbers in a cluster could be affected with a cluster 
sample size this small. Also, the variance of the estimates with the modified method will be 
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20 to 30 percent larger than with the standard method since trimming is not very effective with 
small sample size. Therefore, in most surveys with sample sizes of less than 10 numbers per 
cluster, the problems of implementing the standard method should be quite serious before a 
decision is made to abandon it and use the modified method. 
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The Blaise System for Integrated Survey Processing 


JELKE G. BETHLEHEM and WOUTER J. KELLER! 


ABSTRACT 


Application of recent developments in computer technology allow national statistical offices to produce 
high quality statistics in an efficient way. At the Netherlands Central Bureau of Statistics (CBS) an 
increasing use is made of microcomputers in all steps of the statistical production process. This paper 
discusses the role of software and hardware in data collection, data editing, tabulation, and analysis. 
To avoid the negative effects of uncontrolled de-centralized data processing, the importance of integration 
is stressed. This makes the statistical production process easier to manage, and moreover it increases its 
efficiency. The Blaise System, developed by the CBS, is discussed as a data processing tool that encourages 
integration. Using a description of the survey questionnaire, this system is able to automatically generate 
various computer programs for data collection (CAPI or CAT]I), or data entry and data editing (CADI). 
The system can also create interfaces to other packages. Particularly, the link between Blaise and the 
internally developed packages Bascula (for weighting) and Abacus (for tabulation) is described. In this 
way the Blaise System controls and co-ordinates, and therefore integrates, a large part of the survey process. 


KEY WORDS: Integration; Survey processing; CAPI; CATI; Microcomputers; Decentralization; 
Standardization. 


1. INTRODUCTION 


The Netherlands Central Bureau of Statistics (CBS) makes an increasing use of micro- 
computers in survey data processing. The introduction of microcomputers has a considerable 
impact on the way the work of the statistical office is carried out. Subject matter statisticians 
become increasingly aware of the potential of the new technology, and consequently use it more 
and more in their daily work. 

This paper discusses the role of the new automation technology in data collection, data 
editing, tabulation, and analysis. We will stress the importance of standardization and integra- 
tion. These working policies have three advantages: they enable us to avoid the negative effects 
of uncontrolled de-centralized data processing, they make the statistical production process 
easier to manage, and they increase efficiency. 

The Blaise System, developed by the CBS, is discussed as the backbone of an integrated 
survey processing system. On the one hand, the power of this system lies in the consistency 
it enforces in the various steps of data collection and data processing. On the other hand, it 
also promotes standardization between different departments. Since all departments use the 
same software for processing their surveys, everybody speaks the same ‘‘language’’, and so 
exchange of information between departments is easier and less error prone. 


2. THE STATISTICAL PRODUCTION PROCESS 


National statistical offices collect data on persons, households and establishments and 
transform this information into useful statistics. Production of statistical information is often 
a complex, costly and time-consuming process. This section describes the various steps the 
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Table 1 
The statistical production process 


Design of the survey 
Data editing 


Weighting adjustment 
Tabulation, Analysis 


statistical office has to go through, the problems that it may encounter, and the decisions it 
has to make. An overview of the process is given in table 1. 

The first step is, of course, the design of the survey, in which the statistician specifies the 
population to be investigated, the data to be collected, and the characteristics to be estimated. 
Since statistical offices collect most data by means of (sample) surveys, a questionnaire has 
to be defined, containing the questions to be asked of the respondents. This questionnaire is 
the first practical description of the data to be collected. Furthermore, in the case of a sample 
survey, the statistician also has to specify a sampling design, and he must see to it that the sample 
is selected properly. 

The second step in the process is data collection. Traditionally, in many surveys the ques- 
tionnaires are completed in face-to-face interviews: interviewers visit respondents, ask ques- 
tions, and record the answers on (paper) forms. The quality of the collected data tends to be 
good. However, since it typically requires a large number of interviewers, who may all have 
to do much travelling, it can be expensive and time-consuming. Therefore telephone inter- 
viewing is sometimes used as an alternative. The interviewers call the respondents from the 
statistical office, and thus no more travelling is necessary. However, telephone interviewing 
is not always feasible: only connected people can be contacted, and the questionnaire should 
not be too long nor too complicated. A mail survey is cheaper still: no interviewers at all are 
needed. Questionnaires are mailed to potential respondents with the request to return the com- 
pleted forms. Although reminders can be sent, the persuasive power of the interviewer is lacking, 
and therefore response tends to be lower in this type of survey, and so does the quality of 
collected data. 

If the data are collected by means of paper forms, completed questionnaires have to undergo 
extensive treatment. In order to produce high quality statistics, it is vital to remove any errors. 
This step is called data editing. Three types of errors can be distinguished: A range error occurs 
if a given answer is outside the valid set of answers, e.g. an age of 348 years. A consistency 
error indicates an inconsistency in the answers to a set of questions. An age of 8 years may 
be valid, and a marital status ‘‘married’”’ is not uncommon, but if both answers are given by 
the same person, at least in the Netherlands, there is something definitely wrong. The third 
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type of error is the routing error. This type of error occurs if the interviewer or the respondent 
fails to follow the specified branch or skip instructions, i.e. the route through the questionnaire 
is incorrect: irrelevant questions are answered, or relevant questions are left unanswered. 

Detected errors have to be corrected, but this can be very difficult if it has to be done after- 
wards, at the office. In many cases, particularly for household surveys, respondents cannot 
be contacted again, so other ways have to be found to do something about the problem. 
Sometimes it is possible to determine a reasonable approximation of a correct value by means 
of an imputation technique, but in other cases an incorrect value is replaced by the special code 
indicating the value is ‘‘unknown’’. 

In addition to data editing, another activity is sometimes carried out during this stage of 
the production process: coding of open answers. A typical example is the question about the 
occupation of the respondent. Questions are easiest to process if a respondent selects one 
possibility from a list of pre-coded answers. However, for a question like occupation this set 
of pre-coded answers would be very long, and thus it would be very hard for the respondent 
to select the proper answer. This problem is avoided by letting the respondent formulate his 
own answer, and then literally copying the answer on the form. To enable analysis of this type 
of information, answers must be classified afterwards. This is a time-consuming and costtly 
job, which must be carried out by experienced subject-matter specialists. 

After data editing, the result is a ‘‘clean’’ file, i.e. a file without errors. However, this file 
is not yet ready for tabulation and analysis. In the first place, the sample is sometimes selected 
with unequal probabilities, e.g. establishments are selected with probabilities proportional to 
their size. The reason is that a clever choice of selection probabilities makes it possible to produce 
more accurate estimates of population parameters, but only in combination with an estimation 
procedure which corrects for this inequality. In the second place, representativity may be 
affected by nonresponse, i.e. for some elements in the sample the required information is not 
obtained. If nonrespondents behave differently with respect to the population characteristics 
to be investigated, the results will be biased. 

In order to correct for unequal selection probabilities and nonresponse, a weighting adjust- 
ment procedure is often carried out. Every record is assigned some weight. These weights are 
computed in such a way that the weighted sample distribution of characteristics like sex, age, 
marital status and area reflects the known distribution of these characteristics in the population. 

In the case of item non-response, i.e. answers are missing on some questions, and not all 
questions, an imputation procedure can also be carried out. Using some kind of model, an 
estimate for a missing value is computed and substituted in the record. 

Finally, we have a clean file which is ready for analysis. The first step in the analysis phase 
will nearly always be tabulation of the basic characteristics. Constructing a table is not as simple 
as it may look at first sight. The composition of rows and columns (often built from a number 
of variables), the quantities displayed in cells (counts, means, percentages), the way in which 
percentages are computed, treatment of multiple-response variables, the position of totals and 
subtotals, and many other things, can make life very difficult. 

Many statistical offices also carry out analysis on their data in order to reveal the underlying 
structures, and thus to gain insight in the data. Information obtained in this way may improve 
a later survey, and thus improve quality or reduce costs. 

The results of the analysis will be published in some kind of report. Usually it will contain 
tables and graphs. It is important to present the statistical information in such a way that the 
proper ‘‘message’’ is conveyed. Graphs, tables and text should be simple and clear. Particular 
attention should be paid to graphs, because a visually ambiguous or confusing graph will quite 
easily lead to wrong interpretation. 
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3. THE NEED FOR INTEGRATION 


The computer has always been important in statistical information processing. In the 
beginning to computer was only used for activities like sorting, counting and tabulation. In 
the sixties and seventies, with the emergence of mainframes and statistical packages, it become 
possible to carry out extensive analysis. The computer was also increasingly used for data 
editing, weighting adjustment and imputation. The use of computers for data collection is more 
recent. This first occurred for telephone interviewing (CATI). In the last decade, the advent 
of the small laptop computers has made it possible for interviewers to take the computer with 
them to the homes of the respondents. This way of computer assisted face-to-face interviewing 
is denoted by CAPI. 

It will be clear that the computer is used for more and more activities. Hardware and software 
are available for nearly every step in the production process. Also an increasing number of 
people are making use of the automation tools. At first, only the computer specialists had access 
to ‘‘their’’ machines, but now statisticians and subject matter experts have become computer- 
oriented, and therefore make increasing demands for suitable software and hardware to do 
their jobs. Simple and straightforward electronic data processing can, and is, now carried out 
by the subject matter departments themselves, leaving design and maintenance of complex infor- 
mation systems to be carried out by the computer specialists of the automation department. 
As aconsequence of these developments the work of the statisticians and subject-matter experts 
have changed. They used to be specialists in their own (narrow) field, but now they have 
acquired more general knowledge and experience in a much broader field containing subject 
matter aspects, statistical methodology and computer processing. So the specialists have 
vanished, and a new group with general knowledge of all aspects of survey processing has 
emerged. 

Automation of the statistical production process is nice, but one should be aware of the 
dangers. Although application of computers promises increased efficiency and quality, an 
uncontrolled and unco-ordinated use of the new technology may easily lead to chaos, and 
hence to less productivity. Factors affecting the efficiency of the statistical production process 
are: 


- Different departments are involved. 
Many people deal with the information: respondents fill in forms, subject-matter specialists 
check forms and correct errors, data typists enter the data in the computer, and programmers 
construct editing programs. Transfer of material from one person/department to another 
can be a source of error, misunderstanding and delay. 


- Different computer systems are involved. 
Various data processing activities may be carried out on different computer systems. Transfer 
of files causes delay, and incorrect specification and documentation may produce errors. 


- Repeated specification of the data. 

In almost every step of the process, the structure of the data must be specified. The particular 
system or department has to know about the data: What is the meaning of the variables? 
Which values are permitted? Are there any constraints on the routing? Which relationships 
between variables have to be checked? Although essentially the same, the form of specification 
may be completely different for every step. Every system uses its own ‘‘language’’. The first 
specification is the questionnaire itself. Another specification may be needed for data-entry, 
and yet another for the checking program, for tabulation and analysis, efc. It is clear that 
this is not the most efficient way to deal with the information. 
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The CBS solution to these problems is integration. In this context, integration has three 
different aspects: integration of work, hardware, and software. Let us first have a look at 
integration of the work. 

Traditional data processing consist of what we call macro cycles. All survey data as a whole 
goes through cycles: from one department to another, and from one computer system to 
another. First the paper forms are cleaned manually by the subject matter department, then 
data on the forms are entered by the data entry department, next the files are transfered to 
a mainframe computer system. A program checks the data for consistency, detected errors 
are printed on lists that are send back to the subject matter department for corrections. This 
process of data entering and data editing has to be repeated a number of times before the data 
can considered to be ‘‘clean’’. 

The idea behind integration of work is that the macro cycles should be replaced by micro 
cycles. Not the whole data file, but instead only one record at a time should cycle around. Micro 
cycles means that cycling should take place within one computer system, and that this should 
be controlled by one department. Going from macro cycles to micro cycles comes down to con- 
centrating all data processing activities in one department, and that is the subject-matter depart- 
ment. Since the subject-matter statisticians are the ones with most knowledge about the area 
covered by a survey, they are best equipt to deal with the data, to solve problems, and to produce 
high quality statistics. Of course, they need proper instruments to do their job, i.e. powerful 
and user-friendly software and hardware. 

The idea that automation of data processing activities should be carried out exclusively by 
computer specialists is out of date. More and more the subject-matter statisticians become aware 
of the possibilities and usefulness of the computer for their own work. So the time has come 
for subject-matter departments to take simple and straightforward survey data processing into 
their own hands. Of course, the automation department is responsible for providing the proper 
automation infrastructure. And this department stays in charge of design and maintenance 
of complex information systems. 

The second aspect of integration is integration of hardware. The idea is to concentrate work 
on one type of computer as much as possible. Taking into account that a large number of inex- 
perienced statisticians will have to use the computer, the obvious choice is the microcomputer. 
Microcomputers offer user-friendliness at a relative low price, and moreover, there is an abun- 
dance of useful software. 

Being aware of the fact that statistical offices process huge quantities of data, one may 
wonder whether microcomputers have the capacity to carry out all work, and indeed can take 
over from the large workhorses, the mainframes. To be able to answer this question, it is useful 
to distinguish between two kind of data processing activities. In the first place there are record 
oriented activities. These are activities for which only one record at a time is needed. Examples 
of record oriented activities are data entry and data editing. Record oriented activities are 
generally very well suited for interactive processing. In the second place, there are file oriented 
activities. These activities can only be carried out properly if the whole file is available. Examples 
are the computation of weights and tabulation. Because of their size, file oriented activities 
are often processed in a batch-wise fashion. 

The viewpoint of a few years ago was that record oriented activies could be carried out on 
microcomputers but file oriented activites had to take place on mainframes. With the increasing 
power of microcomputers, attention is shifting in the direction of the microcomputer. At this 
moment, the policy of the CBS is that all record oriented activities have to be carried out on 
microcomputers and file oriented activities can in many cases (say, with data files of less than 
50 megabytes) also be carried on microcomputers. However, for data storage and large batch 
jobs we still need mainframes. 
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The users of the computer environment should be confronted as little as possible with the 
mainframe. Therefore, the CBS is moving in the direction of front end/back end systems. The 
front end consists of microcomputers, and that is what the statisticians use to specify their 
problems. The back end is a mainframe or mini-computer, and is used bulk work, maybe even 
without the user knowing it. Particularly for database applications the client/server approach 
looks very promising. In this approach, the real database activities take place on a dedicated 
minicomputer, whereas the activities are specified, initiated and controlled by the micro- 
computers at the desks of the users. 


4. STANDARDIZATION 


The CBS makes an increasing use of microcomputers (running under MS-DOS) in many 
steps of the statistical production process. On the one hand, this opens new ways towards 
efficient information processing, but on the other hand, it creates new problems that have to 
be dealt with. If every department is free to select and purchase its own type of computer and 
software, the automation infrastructure may easily get out of control, and turn into chaos. 
Departments will not talk the same ‘‘language’’ anymore, because they use different data 
formats and different software. It is clear that this calls for a strong policy on standardization. 
The CBS has adopted such a policy, and in practice it means that there are only one or two 
software packages available for a particular task. 

Another advantage of standardization is that it limits the amount of training that has to 
be provided for the users. In order to cope with the problem of training a large number of new 
microcomputer users, the CBS runs an average of 50 one-day courses per month (occupying 
three fully equipped lecture rooms every working day). 

Attention should also be paid to the way in which the microcomputers are used in the 
organization. Distribution of a lot of stand-alone microcomputers may seem a simple solution, 
but there are also problems that have to be solved. In the first place, it is very easy to copy 
(confidential) data files on local hard disks, so we have a data security problem. Furthermore, 
activities like making back-ups and archiving are often neglected by the users in the subject- 
matter departments. Also communication between departments (e.g. sharing data files) is only 
possible by exchanging floppy disks. Finally, distribution of new releases of software packages, 
including their documentation, is often cumbersome in large organizations with a lot of stand- 
alone microcomputers. 

To avoid the above mentioned problems, the CBS has installed approximately 60 local area 
networks (LANs). Every department has its own LAN. Ten to sixty microcomputers are 
connected to a high-end 386-based fileserver with a storage capacity of up to 600 Megabytes. 
In this environment there are in total nearly 2,300 microcomputers, half of them based on the 
Intel 386SX micro-processor. Security is guaranteed by means of password protection in a login- 
procedure, by encryption, and by using floppy-less workstations (of the 2,300 microcomputers 
only 60 have a floppy or hard disk drive). Archiving and backing-up the LANs is carried out 
in a centralized way by the automation department. A full backup of more than 15 Gigabytes 
is carried out every night. It is clear that version control and updating software can more easily 
be realized in such an environment. Distribution and installation of new software releases on 
a LAN is easy, since, with one command one can upload the new version to all fileservers. All 
software licenses are based on concurrent usage, which is checked by home-made software. 

The role of microcomputers in the statistical production process is growing, but for the time 
being, there are still applications (like the use of large databases) that need mainframe or 
minicomputer systems. In this environment, the CBS has adopted Oracle as the standard 
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database system. Development of a database application is preferably carried out on a 
microcomputer, whereas actually running it takes place on a mini computer. Recently, the CBS 
realized a client/server architecture based on a distributed database system. Microcomputers 
in the network serve as front ends and the minicomputers as back ends. 

So, as the use of the data processing instruments is brought closer to the subject-matter 
specialists at the departments (de-centralization), standardization and coordination of the work 
environment of the subject-matter users demands strong centralization. More details about 
the automation infrastructure can be found in Keller, Metz and Bethlehem (1990). 


5. INTEGRATION OF THE SURVEY PROCESS 


The previous section discussed the need for integration in the survey process. Particular atten- 
tion was paid to concentrating the work in subject-matter departments, and standardization 
of the hardware and software instruments. But standardization of software is not enough. The 
efficiency of the production process can be increased even more by integrating the required 
standard software into one system. This section describes how such an integrated system for 
survey processing is implemented at the CBS. 

An integrated system for survey processing should be based on a powerful language for the 
specification of questionnaires. This specification is the ‘‘knowledge base’’, containing all 
knowledge about the questionnaire and the data. The system should be able to exploit this 
knowledge, i.e. it must be able to automatically generate all required data processing applica- 
tions. On the one hand it means the automatic generation of software for data collection, data 
entry and data editing, and on the other hand the automatic generation interfaces for other 
data processing software, e.g. for tabulation and analysis. In this way repeated data specifica- 
tion is no longer necessary, and consistency is enforced in all data processing steps. 

The backbone of the integrated survey processing system developed by the CBS is the Blaise 
System. In the design phase of the survey, the questionnaire is specified in the Blaise language. 
And it is this specification that is used throughout the whole survey process to extract the infor- 
mation necessary to carry out the various data processing steps. Table 2 summarizes the 
integrated system for survey processing. 

The Blaise System can produce three kinds of programs: CADI, CAPI and CATI programs. 
CADI stands for Computer Assisted Data Input. It integrates data entry and data editing by 
offering an interactive environment for processing paper questionnaire forms. The Blaise 
System can also produce the software required to carry out CAPI or CATI interviewing. The 
Blaise System is discussed in more detail in section 6. 

Whatever form of data collection is used, the result will be a ‘‘clean’’ data file, i.e. a file 
in which no more errors can be detected. The next step in the process will often be the com- 
putation of adjustment weights. The program Bascula will take care of this. It is able to read 
the Blaise data files directly, and extract the information about the variables, i.e. the meta- 
information, from the Blaise specification. Running Bascula will cause an extra variable to 
be added to the data file containing the adjustment weight for each case. More about Bascula 
can be found in section 7. 

Now the file is ready for tabulation, and for that, the integrated system offers the program 
Abacus. This program is also able to read and understand the data files created in the previous 
step of the process. See section 8 for details. Tabulation may be followed by a more extensive 
analysis of the data. For that purpose the Blaise System can generate interfaces for the statistical 
packages SPSS and Stata. More about this in section 9. 
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Table 2 
Integrated survey processing 


BLAISE 


Questionnaire Specification 


CAPI CATI PAPI 


CADI 


Weighting: Bascula 


Tabulation: Abacus 


Analysis: SPSS/Stata 


Publication: PC-Write 


Finally, a publication will be prepared using the standard wordprocessor PC-Write. Since 
this wordprocessor runs on the same computer system as the other software, it is easy to import 
generated tables and results of statistical analysis into the text. 


6. THE BLAISE SYSTEM 


The Blaise System was developed by the CBS, and it derives its name from the famous French 
theologian and mathematician Blaise Pascal (1623-1662). The basis of the Blaise System is the 
Blaise language, which is used to create a formal specification of the structure and contents 
of the questionnaire. The Blaise language has its roots, in large part, in the programming 
language Pascal. 

The Blaise System runs on microcomputers (or networks of microcomputers) under MS-DOS. 
It is the backbone of the integrated survey processing system, and as it is intended to be used 
by the people of the subject-matter departments, one need not be a computer expert to use 
the Blaise System. The design goal of the system was to provide subject-matter experts with 
a powerful but user-friendly tool that enables them to input their knowledge about a survey 
into the system, and to take care of all subsequent data processing steps. 

In the Blaise philosophy, the first step in carrying out a survey is to design a questionnaire 
in the Blaise language. Such a specification of the questionnaire contains more information 
than a traditional paper questionnaire. It not only describes questions, possible answers, and 
conditions on the route through the questionnaire, but also relationships between answers that 
have to be checked. 
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The Blaise System can produce programs for CADI, CAPI or CATI. A CADI program 
is an intelligent and interactive system for data entry and data editing of data collected by means 
of paper forms. The subject-matter specialist works through a number of forms with a 
microcomputer, processing them one-by-one. He enters answers to questions at the proper 
places and, after completion of the form, he activates the check option to test routing and con- 
sistency. Detected errors are reported and explained on the screen. Errors can be corrected by 
consulting the form or calling the supplier of the information. After elimination of all errors, 
a clean record is written to a file. 

The CADI program can also be used for a different way of data processing not mentioned 
thus far. Sometimes statistical offices do not carry out their own data collection, but they have 
to create statistics using data files that were generated elsewhere, outside the statistical office. 
In these cases the data still has to be checked. The Blaise System has a facility to import this 
kind of data files. With a CADI program, an integral check can be carried out on all records 
in a batch-wise version. Thus the records are assigned either the status ‘‘clean’’ or ‘‘dirty’’. 
And the dirty records can be corrected interactively, again with the CADI program. 

A CAPI/CATI program can be used for computer assisted interviewing. The paper ques- 
tionnaire form is replaced by a computer program containing the questions to be asked. This 
computer program is in control of the interview. It determines the proper next question to be 
asked, and checks the answers as soon as they have been entered. In the case of CAPI, the 
interviewing program is loaded into a laptop computer, and the interviewer takes this computer 
to the homes of the respondents. In the case of CATI, the program is in a desktop computer. 
The interviewer calls the respondents from a central unit, and carries out the interview by 
telephone. 

The generation of a Blaise CADI/CAPI/CATI proceeds in a number of steps. First, a text 
editor is used to enter the Blaise specification of the questionnaire, after which it is checked 
for syntax errors. Detected errors must be corrected, and to do that the system returns to the 
text editor and places the cursor on the approximate location of the error. After correction, 
the specification is checked again. If no errors are detected, the specification is transformed 
into Pascal source code, which in turn is compiled into an executable program. 

The Blaise language must serve two somewhat conflicting purposes. On the one hand it must 
be powerful enough to be able to deal with all kinds of large and complex surveys, and on the 
other, Blaise questionnaire specifications must be readable enough, for use by subject matter 
specialists. In fact, a Blaise questionnaire must be self-documenting, i.e. it is the basic descrip- 
tion of the survey which can be used by all people involved. Table 3 gives an example of a simple 
questionnaire in Blaise. 

The first part of the questionnaire specification is the QUEST section, containing the defini- 
tion of all questions that can be asked. A question consists of an identifying name (for internal 
use in the questionnaire), the text of the question as presented to the respondents, and a 
specification of valid answers. The next part of this sample Blaise questionnaire is the ROUTE 
section. It describes under which conditions, and in which order the questions have to be asked. 
Consistency checks are specified in the CHECK section. 

The description above does not exhaust the power of the Blaise language. An overview of 
the Blaise language can be found in Bethlehem ef a/. (1989b), and more details in Bethlehem 
et al. (1989c). 

The Blaise System contains a module for interactive coding, thereby providing the possibility 
of integrating coding either in the data collection phase or in the data entry and data editing 
phase. The module contains two different tools. The first tool implements a hierarchical 
approach to coding. Coding of an answer starts by entering the first digit of the code by selecting 
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Table 3 
A simple Blaise questionnaire 


QUESTIONNAIRE Work ‘‘The Work Survey’’; 


QUEST 
SeqNum  ‘‘Sequence number of the interview?’’: 1..1000 (KEY); 
Age ‘‘What is your age?’’: 0..99; 
Sex ‘*Are you male or female?’’: (Male, Female); 
MarStat ‘‘What is your marital status?’’: 
(Married ‘‘Married’’, 
NotMar ‘‘Not married’’) 
Job *‘Do you have a job?’’: (Yes, No); 
JobDes ‘‘What kind of job do you have?’’: STRING [20]; 
Income ‘*What is your yearly income?’’: 
(Less20 ‘‘Less than 20,000’’, 
Upto40 = ‘‘Between 20,000 and 40,000’’, 
More40 = ‘‘More than 40,000’’); 
Travel ‘‘How do you usually travel to your work?”’: 


SET [3],O5 
(Walking ‘‘Walking’’, 
Bicycle ‘‘By bicycle’, 


Car ‘*By car or motorcycle’’, 
PubTrans ‘‘By bus, tram, train or metro’’, 
Other ‘‘Other means of transport’’); 


OthTrans ‘‘What other means of transport?’’: STRING[20]; 


ROUTE 
SeqNum; Age; Sex; MarStat; Job; 
IF Job = Yes. THEN 
JobDes; Income; Travel; 
IF Other in Travel THEN OthTrans ENDIF 
ENDIF 


CHECK 
IF Age < 15 ‘‘Respondent is younger than 15’? THEN 
MarStat = NotMar ‘‘he/she is too young to be married !”’ 
ENDIF 


ENDQUESTIONNAIRE. 


the proper category from a menu. After the user enters a digit, the program presents a sub- 
sequent menu containing a refinement of the previously selected category. So the description 
becomes more and more detailed until the final digit is reached. The second tool consists of 
a dictionary approach to coding. It tries to locate an entered description in an alphabetically 
ordered list. If the description is not found, the list is displayed, starting at a point as close 
as possible to the entered description. The list can be made so that almost any description, 
including permutations, is present. The advantage of this method is that it is simple, fast and 
controllable. Both coding tools can be used simultaneously. 
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7. BASCULA 


The clean file with sample survey data produced by the Blaise System is usually not ready 
yet for making inference about the population from which the sample has been drawn. The 
problem is that the data do not constitute a representative sample, and so some adjustment 
procedure has to be carried out. 

In order to account for unequal selection probabilities and nonresponse, one often has to 
compute adjustment weights. Post-stratification is a well-known technique. Every record is 
assigned some weight, and these weights are computed in such a way that the weighted sample 
distribution of characteristics like sex, age, marital status, and area reflects the known distri- 
bution of these characteristics in the population. Two major problems can make application 
of post-stratification difficult: empty strata and lack of adequate population information. 
Research has been carried out at the CBS in order to improve weighting techniques. The result 
was a new general method for weighting, in which weights are obtained from a linear model 
which relates the target variables of a survey to auxiliary variables. Post-stratification is a special 
case of this method. Because of the generality of the method, different weighting schemes can 
be applied that take advantage of the available population information as much as possible, 
and at the same time avoid the above mentioned problems. See Bethlehem and Keller (1987) 
for more details. 

Bascula is a general weighting program, running on microcomputers under MS-DOS. It 
combines several weighting techniques. In the first place, traditional post-stratification can 
be carried out. And if the number of empty strata is small, one can instruct the program to 
collapse (i.e. combine) these strata with neighbouring strata. In the case of many empty strata, 
or lack of sufficient population information, Bascula can carry out the linear weighting 
technique described above or apply iterative proportional fitting (also called multiplicative 
weighting, or raking ratio estimation). The resulting weights can either be added to the data 
file, or be stored in a separate file. 

Bascula is able to read the Blaise data file directly, and also extracts the required information 
about the variables from the Blaise specification. The information about the population has 
to be provided by the user. The program is menu-driven, making it user-friendly. It will carry 
out a complete post-stratification if possible. If not, the user has to decide either to carry out 
linear or multiplicative weighting. 

Presently, Bascula can only be used on a microcomputer. In the future, a back end will be 
developed that will run on our mainframe environment. Bascula was particularly developed 
for use in social and demographic surveys, where post-stratification is combined with relatively 
simple estimation procedures. For use in economic surveys, different software will be devel- 
oped. This software will concentrate on stratified sampling designs in combination with more 
complex estimators (ratio and regression estimation). 


8. ABACUS 


Tabulation is one of the basic activities in the statistical production process, and it was one 
of the first to be automated. Many tabulation packages have already been developed in the 
world, but many are not very user-friendly. This is partly caused by the fact that proper genera- 
tion of a complex table needs a lot of parameters to be specified: the variables to be used in 
the various dimensions (rows, columns, layers), whether to concatenate variables (display all 
values of a variable, followed by all values of another variable) or to nest variables (display 
for every value of one variable all possible values of another variable) within a dimension, the 
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displayed cell quantity (counts, percentages, totals, averages), whether to display totals and 
subtotals, and many layout features. To be able to cope with all these parameters, traditional 
packages have control languages to specify tables, and these languages are often not very easy 
to learn and to use. 

Abacus is a tabulation package, running on microcomputers under MS-DOS. While Abacus 
may be seen as yet another tabulation package, it was developed with very specific design goals. 
In the first place, no control language is used to specify a table. The program is menu-driven 
instead. The user designs his table in an interactive, simple, and intuitive way, without having 
to know about any control language. In the second place, Abacus can directly read the data 
file created by the Blaise System, as well as Ascii files. The meta-information, i.e. the infor- 
mation about the variables in the file, can be generated by the Blaise System, or can (in the 
case of separate Ascii files) be entered interactively by the user. Thirdly, the program can pro- 
duce camera-ready tables. 

Another striking property of Abacus is its speed. A table produced by SPSS-Tables in 3 
minutes was generated by Abacus in about 6 seconds (all timings based on the same 386SX 
based microcomputer). The reason for this is that the Abacus program is rather small, so it can 
use a large part of the memory as a working area which allows for a table of up to 90,000 cells. 

Tables produced by Abacus can have up to three dimensions (layers, rows and columns). 
Every dimension can hold up to 10 variables, which may be nested or concatenated. In the 
example in table 4, the column variables ‘‘Employment’’ and ‘‘Sex’’ are concatenated while 
in the row dimension the variables ‘‘Region’’ and ‘‘Town’’ are nested. In this example no 
variable has been placed in the layer dimension. 

This table contains simple counts, but Abacus can also calculate totals of quantitative 
variables, make percentages tables, and averages tables. It is also possible to have more than 
one (up to 10) items in the cells of the table. In that case, the user has to decide to put each item 
in a separate row, column or layer. If the data has been collected by means of a sample survey, 
Abacus can accommodate weighted data, using the weights that are, for example, computed 
by Bascula. The only thing the user has to do is to specify the variable containing the weights. 


Table 4 


An example of a two-dimensional table 


The population of Samplonia 


Number of 
Reis sees Employment Sex 
Job No Job Male Female 

Total 1,000 341 659 Sil 489 

Agria 293 121 ilg/92 145 148 
Wheaton 144 60 84 70 74 
Greenham 94 38 56 44 50 
Newbay 55 23 32 31 24 

Induston 707 220 487 366 341 
Oakdale 61 26 35 36 25 
Crowdon 244 73 171 128 116 
Smokeley 147 49 98 80 67 


Mudwater 255 72 183 122 133 


Source: Samplonian Statistical Office. 
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Much attention has been paid to the layout of the table, because the tables produced should 
be camera-ready. Therefore there are many options in Abacus to control the layout. It is possible 
to specify up to 10 lines of text for the header and for the footer of the table, and one can select 
both horizontal and vertical rules (as in the example), only horizontal rules or no rules at all. 
The layout of the text in column headers and the width of the columns can also be influenced. 

A rounding procedure can be carried out to protect confidential data in the table. Cell totals, 
but also marginal totals are rounded to a multiple of some specified constant, e.g. 5. Abacus 
can provide both normal rounding and random rounding. If the user is not satisfied with the 
resulting table, he can import the output of Abacus into the spreadsheet program Lotus 123, 
and carry out further processing there. A final feature to be mentioned here is the possibility 
of creating new variables by recoding existing variables (e.g. from age to age classes). More 
details about Abacus can be found in Bethlehem ef al. (1989a). 


9. ANALYSIS 


The CBS has not developed any software for statistical data analysis, the main reason being 
that there are already enough good statistical packages available. The CBS itself uses the 
packages SPSS (both on mainframe and micro) and Stata (on micro). To make these packages 
part of the integrated system for survey processing, tools have to be available to export the 
data from Blaise to them. This is realized in two steps. First, the data file is converted from 
the Blaise format to Ascii format, and second, the information about the variables, as available 
in the Blaise questionnaire, is translated in such a way that it can be understood by the par- 
ticular package. Thus, a setup file is created. By running this setup from within the statistical 
package, a system file is created. And by loading the system file, the user can start straight 
away with his analysis, without having to bother about specifying the variables, labels, etc. 

The procedure above only works for SPSS and Stata, and not for other packages. Of course, 
this approach could be implemented for every known statistical package, but that would require 
a large programming effort. Instead, a different road was taken. The Blaise System has a special 
setup generator utility. The user ‘‘paints’’ the structure of the setup file in a word processor, 
and by running the setup generator with this general setup description and the Blaise question- 
naire as input, a real setup file is created. So, with the setup generator the user can generate 
setup files for his own favourite package. 


10. CONCLUSION 


The advent of the microcomputer has had a considerable impact on the work of the national 
statistical office. The subject matter statistician is making use of it more and more, and for 
his work, he needs an integrated survey processing system like the one based on the Blaise 
System. The power of this system lies on the one hand in the consistency it enforces in the various 
steps of data collection and data processing. This makes the whole process easier to manage 
and to control. On the other hand it also encourages standardization between different depart- 
ments. Since all departments use the same software for their surveys, exchange of information 
between departments is easier and less error prone. 

The integrated approach to survey processing was developed with in mind a highly centralized 
organization, like that of the Netherlands Central Bureau of Statistics. In such an organiza- 
tion, this approach can lead to a substantial increase in efficiency. However, not all statistical 
offices have a centralized structure. Particularly in larger countries, data processing is often 


56 Bethlehem and Keller: The Blaise System 


decentralized. Regional offices take care of data processing in their own regions, and the 
resulting data files are sent to the central office. In the central office, the regional files are 
combined into one national file. The integrated approach can also be applied successfully in 
such an environment: the central office develops the Blaise questionnaire, and copies of the 
generated data entry program are sent to each regional office. This ensures consistency of data 
collection and data editing at the regional level. The regional data files will all have the same 
Blaise format, so combining them into one national file will be a simple job using the tools 
of the Blaise System. Since all regional files will be ‘‘clean’’, no further editing will be necessary 
at the national level. The only job for the national office will be to tabulate and analyse, and 
to publish the results. Furthermore, if either the regional offices or the central office can make 
regional publications. 

The Blaise System has been tested and used since the middle of 1986 for a substantial number 
of surveys. The system is developed in close cooperation with the users. Every new version 
contains enhanced features. Abacus has been in use for over a year, and is very popular among 
its users. The Bascula program is still in development. The first prototype has just been released. 

We did not mention the possibility of exporting data and meta-information from the Blaise 
System to the Paradox database package. In the future, a link will also be established between 
Blaise and the Oracle database system. In this way, a client/server architecture can be realized 
for Blaise users. 

At the end of the statistical production chain, there are some aspects of publication that 
still have to be dealt with. In the first place, software will be developed to asses the risk of 
disclosure of confidential (private) information in statistical information to be published. Tools 
will also be offered to protect tables or data files against these risks. Finally, statistical offices 
engage more and more in electronic publication of statistical information, i.e. statistical infor- 
mation on floppy disks, CD-ROM, etc. To help the users of this type of information in selecting 
the subset of information they need, user-friendly software must be made available to them. 
This software is now being developed. 
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ABSTRACT 


Findings from the research and testing of telephone and computer assisted survey methods for household 
surveys are presented, followed by discussion of how these findings will influence the redesign of household 
surveys at Statistics Canada during the 1990’s. Significant emphasis is given in the presentation to the 
Canadian Labour Force Survey. 
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1. INTRODUCTION 


The 1980’s have seen significant changes in survey taking due to advances in technology and 
the development of modern telephone survey methods, and the pace of these changes will likely 
accelerate during the 1990’s. In this paper we will describe the research, testing, and development 
of methods that will form the infrastructure underpinning the data collection activities for 
Statistics Canada’s household surveys during the 1990’s. This research has focused in particular 
on the Canadian Labour Force Survey, and was carried out from 1985 to 1989 with a view to 
identifying improvements to be implemented during the 1991 post-censal redesign of the survey. 


2. RESEARCH AND TESTING PROGRAM FOR LFS 


The Canadian Labour Force Survey (LFS) is the largest household survey conducted by 
Statistics Canada, with a sample size of 62,300 households per month. It follows a rotating 
panel design in which households remain in the sample for six consecutive months, after which 
they are rotated out. It is based on a multi-stage area sample, with a decentralized interviewing 
staff of 1,000 local interviewers located across Canada and reporting to one of five Regional 
Offices. 

Until the early 1970’s, all interviewing was face to face. In 1972 telephone interviewing was 
introduced in large urban areas for follow-up interviews with households after they had received 
a face to face interview during their first month in the sample. In the literature, such telephone 
follow-up is referred to as ‘‘warm telephoning’’, to distinguish it from ‘‘cold telephoning’’ 
where the telephone interview is not preceded by a face to face interview (Groves et al. 1988). 

The warm telephoning was initially restricted to major urban areas due to the frequency 
of party lines in smaller urban and rural areas and concerns this raised about the confiden- 
tiality of the data being collected. However, during the 1981 redesign of the survey, warm 
telephone interviewing was tested for the small urban and rural areas, and it was found 
respondents were willing to be interviewed by telephone, and the procedure had no impact on 
response rates or survey estimates (Choudhry 1984). The extension of telephoning to these areas 
in 1984 resulted in a 10% reduction in the data collection costs for the survey. 
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In 1985, following introduction of a redesigned sample, a research program was started to 
investigate what further improvements in data collection could result from: (i) more use of 
the telephone in collection, (ii) telephone survey methods where the telephone is used both for 
sampling and for data collection, and (iii) Computer Assisted Interviewing (CAI) methods. 

In the study of telephone and CAI methods it was useful to characterize the survey design 
in terms of a number of design factors as follows: 


(i) Mode of collection. As already noted, the current mode of collection is warm telephoning, 
with an initial face to face interview and predominantly telephone interviews in later 
months. Alternative modes of collection include cold telephone interviewing with face to 
face follow-up of telephone nonrespondents, and cold telephoning without face to face 
follow-up. 


(ii) Organization of interviewers. The LFS currently features a local organization, with the 
local interviewers doing a mixture of face to face interviews and telephone interviews from 
their homes. An alternative is a central organization, with interviewers working out of 
one or more central cites - in the case of Statistics Canada, its five Regional Offices across 
the country. A third organizational model is a mixed one where interviewing is done by 
a combination of local and central interviewers. 


(iii) Technology. The current technology for the survey is traditional paper and pencil. The 
alternative technology considered most viable for household surveys is Computer Assisted 
Interviewing. CAI is commonly referred to as CATI (Computer Assisted Telephone Inter- 
viewing) when done centrally, and CAPI (Computer Assisted Personal Interviewing) when 
done locally using portable computers that interviewers use for face to face interviews and 
for telephone interviewing from their homes. 


(iv) Frame and sample design. The survey has been based on an area sampling design since 
its inception in 1945. Alternatives include: telephone frames based on either Random Digit 
Dialing (RDD) methods (Waksberg 1978) or a combined use of RDD and list frames of 
published telephone numbers; other list frames which unlike telephone lists, are concep- 
tually complete; and dual frame methods combining two or more of the above frame 
options. 


In the following sections findings from the research and testing program pertaining to each 
of these design factors are discussed. 


3. MODE OF COLLECTION 


A major test was carried out from 1985 to 1989 to determine the impact of cold telephone 
interviewing with face to face follow-up as an alternative to the current warm telephoning. 
This test was referred to as the Telephone First Interview Test. The test was embedded into 
the ongoing LFS in urban areas of Quebec and Ontario. The methodology is reported fully 
in Drew, Choudhry and Hunter (1988). In brief, newly sampled LFS dwellings were matched 
to lists purchased from the telephone companies on the basis of address information. Match 
rates of 65% were obtained. Test and control samples were selected from the matched dwellings 
such that each test dwelling was paired with a control dwelling within the same sampling unit 
(city block). For the test dwellings, telephone numbers were provided to interviewers, who were 
to attempt a telephone interview, but to use face to face follow-up if necessary. Interviewers 
were unaware of the existence of a control sample, and followed normal procedures for all 
dwellings for which telephone numbers were not provided. 
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Table 1 


Telephone First Interview Test (October 1985 - March 1989) 
Estimate for Test Treatment as Percent of Estimate for Control Treatments 


Ontario Quebec 

Characteristic —--->T —_ > 

Percent t-statistic Percent t-statistic 
Employment 98.5 —1.22 97.2 — 1.64 
Unemployment 96.3 — 0.94 111.8 PAD 
Not in LF 101.1 0.88 98.2 — 0.63 
Pop. 15+ 909.2 — 0.86 98.5 — 1.31 
Pop. in Hhld = 3+ 97.8 — 0.76 94.1 — 1.26 
Pop. in 1 person Hhlds 100.6 0.25 104.1 —0.93 
Pop. in 2 person Hhlds 101.1 0.26 101.0 0.43 
Pop. in 3 person Hhlds 98.6 — 0.56 94.7 — 1.28 
Emp. Male 15-24 95.2 —0.75 88.7 —2.00* 
Emp. Male 25+ 98.5 -1.11 96.7 — 1.54 
Emp. Female 15-24 99.0 —0.11 11135 1.44 
Emp. Female 25 + 99.2 — 0.65 97.1 —1.11 
Unemp. Male 15-24 105.6 0.53 119.0 1:53 
Unemp. Male 25+ 94.5 — 0.80 96.7 —0.31 
Unemp. Female 15-24 99.6 —0.15 119.4 1.30 
Unemp. Female 25 + 90.9 —1.22 123.9 PAA ACE 
Not in LF Male 15-24 99.8 — 0.07 99.9 0.47 
Not in LF Male 25+ 105.6 1357 101.3 0.07 
Not in LF Female 15-24 101.9 0.56 95.3 — 0.56 
Not in LF Female 25 + 99.2 —0.14 97.3 —0.92 
Pop. Male 15-24 97.2 — 0.49 952 —0.73 
Pop. Male 25+ 99.9 — 0.09 97.7 — 1.49 
Pop. Female 15-24 99.9 0.16 105.4 0.95 
Pop. Female 25 + 98.9 — 1.28 98.4 — 1.24 


* t-statistic significant at 5% level 
** t-statistic significant at 1% level 


No significant differences in response rates were found between the test and control samples. 
For Quebec, response rates were 96.1% for both samples, while for Ontario, the rate of 96.3% 
for the test sample was marginally lower than that of 96.5% for the control sample. 

When comparing the labour force estimates obtained in the test and control samples, certain 
estimates from Quebec early in the test for the period October 1985 to February 1987 were 
significantly different. In particular the employed and unemployed males in households of three 
or more persons were underestimated in the test sample (see Drew, Choudhry and Hunter 1988). 
Table 1 presents data over the full life of the test from October 1985 to March 1989. For Quebec, 
a few statistically significant differences existed - which stemmed from the influence of the 
earlier time period. When the data are analyzed from March 1987 onwards, these differences 
are not significant. In Ontario, there were no significant differences. Speculating on the 
differences in Quebec, their co-incidence with a program of inspection of welfare recipients 
carried out by the provincial government suggests that measures external to the survey led to 
a climate in which there was decreased trust of cold telephone interviews. We were fortunate 
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Table 2 


Nonresponse Rates: Warm Telephoning versus Cold Telephoning 
with/without Face to Face Follow-up 


Method Test 1 Test 2 
Warm telephoning with letter (ongoing LFS) 4.1 5.6 
Cold telephoning with letter 6.9 9.8 
Cold telephoning without letter 8.5 ~ 


Test 1: October 1985 - September 1986; Ontario and Quebec 
Test 2: July 1988 - March 1989; Nova Scotia and Alberta 


to have been conducting the test during this period, because the finding that survey results 
obtained under cold telephoning are more subject to external influences than are those obtained 
under warm telephone interviewing will be an important consideration in any decisions on 
extension of telephone interviewing. 

Cold telephoning without face to face follow-up was also studied. Two Telephone Sampling 
Tests were carried out in which the LFS was conducted as a central telephone survey, with inter- 
viewing from the Regional Offices. Nonresponse rates for the two tests and comparable rates 
for the ongoing LFS are presented in Table 2. 

The first test studied two sampling methods - RDD, and a combination of list sampling 
for published numbers and RDD for nonpublished numbers. The list sampling featured 
introductory letters, but the RDD sampling did not. Differences in response rates would seem 
to point to the positive effects on response rates of an advance letter. For both tests, the com- 
parison of warm versus cold telephone interviewing revealed nonresponse rates for cold 
telephoning which were higher at a 5% significance level. The second test was based solely on 
a list sample of published numbers. 

An important issue is the nonresponse bias, if any, resulting from the higher nonresponse 
under cold telephoning without face to face follow-up. As a proxy to these extra non- 
respondents, Laflamme (1990) looked at non-first-month-in-sample households from the 
ongoing LFS who had a telephone, but who were interviewed face to face. He found that size 
of the proxy group, at 3.5% of respondents, was close to the size of the extra nonresponse 
under cold telephoning without face to face follow-up. Further, he found the unemployment 
rate for the proxy group was 12.8%, versus 7.4% for persons in households interviewed by 
telephone. Exclusion of the proxy group from the sample would have lowered the national 
unemployment rate from 8.1% to 7.9%. This is clearly a serious bias, given the accuracy 
required for the LFS national estimates. As the proxy assumption seems a reasonable one, these 
findings raise serious concerns about cold telephone interviewing without face to face follow-up 
for the LFS. 

Table 3 compares unemployment and participation rates for the first telephone sampling 
test with corresponding estimates from telephone households in the LFS. The only estimate 
found to be significantly different at a five percent level from estimates produced for the 
telephone population from the ongoing LFS was the unemployment rate for Quebec for the 
RDD treatment. It is worth noting that the test was carried out at the same time that problems 
emerged with estimates for Quebec in the Telephone First Interview Test. Another point worth 
noting is that while other differences in unemployment rates were not statistically significant, 
the rates for cold telephoning were higher. Other researchers have observed differences in the 
same direction, also without being able to attribute statistical significance to them. These data 
might benefit from a meta analysis. 
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Table 3 


Telephone Sampling Test (October 1985 - September 1986) 
Unemployment and Participation Rates 


Unemployment Participation 
Province Design Rate Rate 
(S.D.) (S.D.) 
Quebec LIST 1233 (0.78) 64.1 (1.08) 
RDD ot pe (0.88) 62.8 (1.28) 
LFS 10.9 (0.27) 63.4 (0.29) 
Ontario List pe (0.59) 69.0 (1.11) 
RDD 7.9 (0.63) 69.0 (1.18) 
LFS 6.9 (0.16) 69.0 (0.20) 


* Significant difference between RDD and LFS Unemployment rates for Quebec 


In summary, the test results showed cold telephoning without face to face follow-up yielded 
higher nonresponse rates than the current warm telephoning method, and while inconclusive, 
there was some evidence that it yielded higher unemployment rates. On the other hand, cold 
telephoning with face to face follow-up, apart from the one period of time in Quebec, was 
found to yield data comparable to that under warm telephoning. 

On the basis of these findings, it was decided to implement cold telephoning with face to 
face follow-up for the LFS apartment frame sample, which constitutes roughly 4% of the overall 
sample. The availability of the telephone number for apartment frame units, it was reasoned, 
would help overcome problems in gaining access to highrise apartment buildings, and allow 
for more attempts to find persons at home than is feasible with face to face interviewing. These 
expectations seem to have been borne out. As reported by Dufour (1990), while first month 
nonresponse rates for the apartment sample continue to be higher than corresponding first 
month rates for the non-apartment sample, the gap has narrowed from a difference of 8.7 
percentage points in the year before the change to a difference of 5.7 percentage points during 
the first five months under the new procedure. 

Another change to the mode of collection for the ongoing LFS was to introduce telephone 
follow-up of the first month in sample households which could not be contacted during an 
initial visit to the dwelling. This procedure was introduced in 1986, and led to a $100,000 per 
year savings in data collection costs. 

The combined effect of the telephone first interview for the apartments, and the telephone 
follow-up for first month nonrespondents has been an increase in the overall telephoning rate 
for the survey from 80% in 1985 to 83% in 1990. 


4. ORGANIZATION OF INTERVIEWING STAFF 


During the testing program, two alternatives to the current local organization of the inter- 
viewing staff were studied. The telephone sampling tests already described considered a ‘‘central’’ 
organization where all of the interviewing was done out of the Regional Offices. Another test 
examined a mixed organization, in which the current warm telephoning mode of collection 
was retained. The test of the mixed organization was carried out from January 1988 to March 
1989 in two Census Metropolitan Areas in which Regional Offices are located - Montreal and 
Halifax. Its primary objective was to measure the cost implications of such a mixed organization. 
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The test methodology consisted of face to face collection by local interviewers for first month 
in sample cases, and telephoning by central interviewing staff working out of the Regional 
Offices for most non-first month cases. Whenever nonresponse follow-up was required for 
households initially assigned to the central interviewers, this was carried out by the local inter- 
viewers. This methodology was initially tested for the Labour Force Survey by Muirhead et al. 
(1975) and has been extensively studied by the United States Bureau of the Census (1987), where 
the centralized interviewing is being done using Computer Assisted Telephone Interviewing 
(CATI). 

One of the complexities of the method was the practice followed for the first half of the 
test of transferring cases requiring nonresponse follow-up from the central to the local inter- 
viewers at the mid-point of the interviewing week. For the second half of the test, this so-called 
re-cycling was restricted to cases where the telephone number was determined to be no longer 
valid. During the first half of the test, nonresponse rates were 8.0% for the test treatment versus 
6.1% for the control procedures corresponding to the decentralized interviewing used for the 
ongoing LFS. The gap narrowed to 7.3% versus 6.7% during the second half. 


From the first telephone sampling test, interviewing costs per household were estimated 
to be $2.72 for central data collection with telephone list sampling, versus $3.53 for RDD 
sampling. The extra costs for RDD methods is due to the time spent in screening for residential 
telephone numbers. These costs include $0.46 per household for long distance charges. This 
amount was estimated based on long distance rates and data on length of calls, since record 
keeping practices in the regional offices did not permit the extraction of actual costs incurred. 
Comparable costs for the ongoing LFS were $4.76 per household for interviewer fees and 
expenses. The test of the mixed organization yielded savings relative to the ongoing LFS of 
$0.78 per household in interviewer fees and expenses. The above cost comparisons do not factor 
cost of office space and equipment into the costs under the centralized and mixed organizations. 
Nor do they consider the costs of transferring documents to and from local interviewers under 
the mixed and local organizations, which under the current paper and pencil technology is 
accomplished by express mailing of documents, but under CAI scenarios would be transmitted 
electronically. 


The mixed organization was considered only for Regional Office cities, as extension beyond 
Regional Office cities would imply greater long distance telephoning. More importantly, the 
sample design for smaller urban and rural areas is clustered so that primary sampling units 
yield sample sizes corresponding to an interviewer assignment. Centralization of the telephone 
portion of the sample would necessitate more clustering of the sample in order to retain a 
sufficient workload for the local interviewers. Also in medium sized urban centres where there 
are currently four to five interviewers, the number of local interviewers under the mixed 
organization would be reduced to one or two, significantly reducing the flexibility to have inter- 
viewers fill in for one and other during vacations and illness. 


Of the three organizational models considered, all had advantages and disadvantages. The 
local organization yielded the lowest nonresponse rates, albeit at the highest per unit data 
collection cost. The mixed organization had marginally higher nonresponse and marginally 
lower costs, and was limited in where it could be applied. In the final analysis, the mixed 
organization was seen as introducing a lot of complexity for at best marginal gain. The central 
organization, which offered substantial savings in data collection costs, resulted in a 68-75% 
increase in nonresponse relative to the local organization. As discussed in section (3), there 
is evidence that this extra nonresponse would introduce a serious nonresponse bias into the 
LFS estimates. Moreover there are concerns that the gap in nonresponse rates attainable under 
local versus central organizations might widen in the future, as increasing exposure to telephone 
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solicitation and increasing availability of telephone screening technology renders the population 
less receptive to telephone interviewing. Such developments favour survey design strategies 
which, although they may allow for flexibility to do telephoning, also allow for face to face 
follow-up wherever needed. The local organization best offers this flexibility. On the basis of 
the above considerations it has been decided to retain the current local organization. 


5. TECHNOLOGY 


Catlin, Ingram, and Hunter (1988) carried out a controlled study comparing CATI and paper 
and pencil interviewing. In the study the LFS questionnaire was administered to RDD samples 
of 1,000 households per month per treatment over a period of nine months. All interviewing 
took place from Statistics Canada headquarters in Ottawa. 

The study was part of a collaborative research effort with the United States Bureau of the 
Census (USBC), and the CATI software used was developed by the USBC. The wording of 
the questionnaires was purposively the same for both treatments. Features unique to the 
CATI treatment were automatic branching, some basic on-line edits, and automated call 
scheduling. 

Three quality improvements were discernable for CATI relative to paper and pencil methods. 
First, the overall rate of edit failures during post-collection data processing was 50% lower 
for CATI. Second, there was a virtual elimination of branching errors under CATI. Impor- 
tantly, this occurred for certain portions of the questionnaire, which, although infrequently 
encountered, have a bearing on determination of labour force status, and which under paper 
and pencil interviewing are subject to high levels of branching errors. Third, the average 
household size reported under CATI was 3% higher, which represents roughly a 50% reduction 
in the underenumeration in the LFS relative to the Census. This improvement seems to stem 
from the enforced probing built into the CATI instrument for additional household members 
and for persons temporarily away. 

Based on these findings, it has been decided that the introduction of Computer Assisted 
Interviewing should be one of the major thrusts of the 1991 post-censal redesign of the LFS. 
Due to the preference for maintaining a local organization of interviewing staff, a CAPI 
implementation is being planned for. 


6. FRAME AND SAMPLE DESIGN 


Telephone Frames 


Telephone coverage and the extent to which characteristics of those without telephones differ 
from characteristics of those with telephones are important factors in the design of telephone 
survey methods - particularly as regards frame strategies. 

In an international review of telephone coverage, Trewin and Lee (1988) found telephone 
coverage in Canada to be one of the highest in the world at 97-98%. As is typical of the situation 
in most countries they surveyed, persons in non-telephone households in Canada tend to have 
lower incomes and higher rates of unemployment. 

Table 4 gives the percentage of non-telephone households in Canada from 1976 to the 
present. Telephone coverage, while already high in 1976 has been steadily edging upwards, 
although it appears to have levelled off over the last few years at around 98.5%. 
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Table 4 
Non-telephone Households by Province (%) 

1976 1981 1985 1987 1990 
Canada S100" 2.4 1.8 5 | (es 
Newfoundland 10.0 6.0 Sel 3.6 1.9 
Prince Edward Island - - - - 2.8 
Nova Scotia ES 4.6 Be 3.2 iS 
New Brunswick 5.8 Da 58 3s PL 02 
Quebec 3.3 Pash 1.6 1.5 1:5 
Ontario OOS 1.9 1.0 1.0 ez, 
Manitoba 4.1 2.3 DI 2.4 ies 
Saskatchewan 3.6 25 D3} 2.4 Pied 
Alberta 3.0 2.4 2.0 1.8 2.0 


British Columbia 4.2 2.8 2.4 13 eS 


Source: Statistics Canada, Estimates from Household Facilities & Equipment Survey 


Table 5 
Labour Force Characteristics by Telephone Status 


: Telephone Unemployment Participation 
Browns Status Rate Rate 
Nova Scotia published 9.0 Tr 

non-published 9.8 70.2 
non-telephone L732 62.3 
Alberta published 6.3 80.7 
non-published 8.2 81.5 
non-telephone 11.1 67.0 


Laflamme (1990) undertook a study comparing characteristics of the non-telephone and 
telephone universes. The study included a breakdown of those with published versus non- 
published numbers, obtained by linking telephone numbers supplied by LFS respondents to 
lists of published telephone numbers. Two provinces were included in the study, Nova Scotia 
and Alberta. For Nova Scotia 9.7% of numbers, and for Alberta 11.2% of numbers were found 
to be non-published. Unemployment and participation rates reported by Laflamme are given 
in Table 5. 

This study replicates findings from earlier studies that the labour force characteristics of 
persons without telephones are very different from those with telephones. The labour force 
characteristics differ but to a lesser extent between persons with published versus non-published 
numbers. 

While the non-telephone population accounts for only 1.0%-1.5% of the population, the 
differences in labour force characteristics are sufficiently large that simply excluding the non- 
telephone population is not a viable option for the LFS, given the accuracy required for the 
national employment and unemployment estimates (coefficients of variation of 0.5% and 2% 
respectively). 
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Another difficulty with telephone frames, particulary for panel surveys is their rapid 
deterioration. Drew, Dick and Switzer (1989) found a 0.5 - 1.0 % rate of additions and deletions 
to the stock of published residential numbers per month. Hence telephone samples cannot 
remain representative of the telephone universe for very long unless they are updated. The 
authors proposed a strategy of updating samples for a panel survey using files of published 
numbers acquired on an ongoing basis from telephone companies. An operational test of 
the procedure over a nine month period was a success. Their procedure applied only to 
published numbers sampled from a list frame, and did not provide a solution to the problem 
of keeping a sample selected using Random Digit Dialing methods up to date over the life of 
a panel. 

Because of the coverage and updating problems with telephone frames, approaches where 
dwellings as opposed to telephone numbers are the samping units are seen as having more 
promise for large scale panel surveys. It is worth noting that the situation can be quite different 
for other surveys. Catlin et al. (1984) showed that coverage biases for general population 
characteristics were less than for labour force characteristics. Further for smaller surveys 
(e.g., those with sample sizes of 10,000 or less) small biases are less important given the larger 
relative sampling errors for these surveys. These findings led to establishment of an RDD 
household survey capacity in 1986. It has been used for numerous one-time surveys and for 
the General Social Survey, which is an annual survey of 10,000 households. 


Area Frame 


As has already been described, it is possible in urban areas to match selected addresses from 
an area frame to telephone lists in order to permit cold telephone interviewing. The experiences 
with the LFS have been that telephone numbers can be obtained in this fashion for approx- 
imately 60% of households. These match rates are based on exact matching after standardiza- 
tion of the address information, and they could be improved through use of record linkage 
methods. With telephoning for a substantial portion of the first month cases, the clustering 
of the sample could be reduced somewhat, but a clustered sample remains a constraint imposed 
by an area frame design. 

It is planned to investigate the feasibility of extending these procedures to rural areas, which 
would entail changing the type of information collected when dwelling lists are created for the 
survey. The information currently collected tends to be descriptive of the physical characteristics 
of the dwelling, whereas to successfully match with lists of telephone subscribers, information 
such as name (often readily available on mail boxes), street name and number, or in their 
absence the rural route number and postal code, would be required. 


Address Register 


Statistics Canada is constructing an Address Register in urban areas of Canada. It will be 
used in the 1991 Census to improve coverage by providing an independent check on the dwelling 
lists created by the Census enumerators (Drew, Royce and van Baaren 1989). The Address 
Register will be a machine readable list of addresses constructed by linkage of various 
administrative data sources, including lists of customers with published numbers purchased 
from telephone companies. During the use of the Address Register in the 1991 Census, its 
coverage will be updated to correspond to that of the 1991 Census. 

It is planned during the 1991 post-censal redesign of the Labour Force Survey to conduct 
studies into use of the Address Register as a frame for household surveys in urban areas. If 
the conclusion is that it should be adopted as a frame, the Address Register will be updated 
on an ongoing basis following the 1991 Census. 
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An advantage of the Address Register as a frame over the area frame is that telephone and 
non-telephone households are known ahead of time. Hence the two can be sampled as separate 
strata - with a reduced amount of clustering for the telephone stratum, for which a significant 
portion of the first month interviews could be done by telephone. The non-telephone stratum 
would include those households with non-published numbers and those households without 
telephones. Evidence from earlier studies showed the refusal rate when cold telephoning 
households with non-published numbers to be 12% compared to 4% for all households. Under 
warm telephoning there is no corresponding increase in the refusal rates for households with 
non-published numbers, and there is a good success rate in converting these households to 
respond by telephone in later months. This finding and the desire to be sensitive to privacy 
concerns of individuals support the face to face interview of such households, which account 
for an estimated 10-15% of numbers. 


Dual Frame 


In urban areas, if the coverage of the Address Register as the sole frame is not adequate, 
a dual frame design in which the Address Register is supplemented by a small area sample will 
be considered. There are different forms the supplementary sample could take. A promising 
option, not involving the expense of building and maintaining both a conventional area frame 
and an Address Register, would be to use an interval approach in which a sample of consecutive 
dwellings on the Address Register would be selected and checked in the field. Any dwellings 
found between the Address Register dwellings constitute a sample of dwellings missing from 
the Address Register. 

Mian (1990) has studied dual frame methods considering a cost and variance optimization 
for the general case where neither of the frames needs to cover the entire universe. This was 
felt to be a practical model since the area frame, while conceptually complete, in practice suffers 
from 3-4% undercoverage relative to the Census, in addition to the 5% of the population which 
is not represented because of nonresponse. Extension of Mian’s model to include a non- 
sampling error component will permit factoring into the optimization what we know or may 
wish to assume about the coverage and nonresponse biases under alternative frame approaches. 
It can be used in the context of dual frames combining the Address Register and area frames 
in urban areas, and combining area and telephone frames in rural areas. 


7. 1991 POST-CENSAL REDESIGN OF LFS 


The Labour Force Survey is redesigned following each decennial population census. 
Redesigns have normally focused on redesign of the sample, but in the 1970’s a major revision 
was carried out encompassing a sample redesign, changes to the questionnaire content, wording 
of questions, and survey outputs, and a major overhaul of the survey processing systems 
including introduction of a network of mini-computers in the regional offices to support survey 
operations and regional data capture. In contrast, redesign efforts during the 1980’s were 
restricted to a sample redesign. 

While decisions on the scope of the 1991 post-censal redesign have yet to be taken, an effort 
falling somewhere between the major revision in the 70’s and minimal redesign in the 80’s 
appears needed. The work on the redesign is at an early planning stage, and is proceeding 
through four sub-projects focusing on: (i) content and questionnaire issues, (ii) modernization 
of the survey processing systems and review of survey outputs, (iii) development, testing, and 
implementation of Computer Assisted Interviewing, and (iv) sample redesign. (Drew et a/. 1991). 
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Sub-projects (iii) and (iv) are those which will be concerned with telephone and CAI methods. 
Current plans for these sub-projects as they relate to the survey design factors described in earlier 
sections of this paper are briefly summarized below. 


Technology and Organization 


Based on the positive findings from testing of Computer Assisted Interviewing for the LFS 
reported by Catlin et a/. (1988), it has been decided to make the adoption of CAI one of the 
major thrusts of the redesign. Moreover, for reasons already discussed, a decision has been 
taken that the current local organization of the interviewing staff should be retained, so that 
the implementation of CAI methods will take the form of Computer Assisted Personal Inter- 
viewing (CAPI). Specifically, local interviewers will be equipped with lightweight portable 
computers that they will carry with them for face to face interviewing, and that they will use 
to conduct telephone interviews from their homes. The mix of face to face and telephone 
interviewing may remain much the same as it is currently - 83% telephone and 17% face to 
face - or it may shift to more telephoning if it is decided to adopt more cold telephone inter- 
viewing of households during their first month in the sample. 

The work plan for the Computer Assisted Interviewing sub-project includes a field test during 
1991 of a touch screen portable computer, and in later years development or acquisition of 
CAI software, a combined test of CAI and questionnaire alternatives, development of on-line 
editing, and an automated version of the interviewer manual embedded into a help screen 
accessible during interviewing. 


Frame and Mode of Collection 


A key research finding was that cold telephone interviewing with face to face follow-up yields 
response rates and labour force estimates comparable to those under the current warm telephone 
collection procedure for the LFS, which features face to face interviewing for the first month 
households in the sample. One exception was observed in Quebec, where, as discussed, for 
a period of time data differences were seen. The importance of face to face follow-up in main- 
taining response rates favours the retention of the dwelling as the sampling unit, and supplying 
the telephone number to interviewers to give them greater flexibility to use both telephone and 
face to face interviewing in obtaining first month interviews. 

In urban areas, both the current area frame and the Address Register as a frame are consistent 
with this approach. In rural areas, as described earlier in the paper, research into the feasibility 
of matching area frame addresses to telephone lists to provide interviewers with telephone 
numbers will be studied. It is also planned to continue to investigate dual frame methods which 
in urban areas might consist of the Address Register and an area frame, and in rural areas an 
area frame and a telephone frame. 


8. SUMMARY 


The current data collection methodology for the Labour Force Survey consists of a face 
to face interview for households during their first month in the sample and predominantly 
telephone interviewing in later months. The introduction of telephone interviewing for the later 
months took place during the 1970’s for major urban areas and during the 1980’s for remaining 
areas. In both instances the introduction of telephone interviewing resulted in significant cost 
savings without any impact on the response rates or survey estimates. Prior to the telephone 
and CAI research and testing program begun in 1985, 80% of LFS interviews were done by 
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telephone. This has moderately increased to 83% through the introduction of telephone follow- 
up for households which could not be contacted during an initial face to face visit, and by sup- 
plying interviewers with telephone numbers for the apartment sample. 

The primary benefit of the research and testing program has been to identify the frame and 
data collection options to pursue during the 1991 post-censal redesign of the survey, including 
the retention of the current local organization of interviewers,the adoption of Computer 
Assisted Personal Interviewing, the retention of frame and sample design approaches in which 
the dwelling is the unit of selection, and the provision of interviewers with telephone numbers 
to permit the flexibility to use a combination of telephone and face to face interviewing to obtain 
first month interviews. 
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Marginal and Approximate Conditional Likelihoods 
for Sampling on Successive Occasions 


D.R. BELLHOUSE! 


ABSTRACT 


Marginal and approximate conditional likelihoods are given for the correlation parameters in a normal 
linear regression model with correlated errors. This general likelihood approach is applied to obtain 
marginal and approximate conditional likelihoods for the correlation parameters in sampling on successive 
occasions under both simple random sampling on each occasion and more complex surveys. 


KEY WORDS: Likelihood inference; Sampling in time; ARMA models; State space models. 


1. INTRODUCTION 


Consider a finite population of N units which may be sampled on k occasions. Let y,; 
denote the measurement on the jth population unit taken on the th occasion; 7 = 1, ..., N 
and t = 1, ..., k. It is assumed that any two units, say j and /’, are independent, but that 
measurements of the same unit across time are correlated. In particular, assume that for any /, 


(Vijs Vajo -++s Veg)” ~ N(p,07Q), (1) 


where 2 is a k X k correlation matrix and where p is the 1 x k vector of fixed means 
(141) 2» -» +s Hx) 2. In view of the explicit model assumption in (1), a model-based approach 
to survey estimation is used in this paper. Based on samples taken over the k occasions, it is 
of interest to estimate (1, 42, ..-, Hz) 7. The form of the model-based estimates (ji), fiz, ..-, 
jx) 7, if obtained by maximum likelihood or generalized least squares, for example, will 
depend on o” and the parameters in . It is therefore necessary to obtain good estimates of 
o” and the parameters in Q. 

The notation of Bellhouse (1989) is used to describe the sampling scheme considered here, 
namely one-level rotation sampling. On any occasion, c rotation groups are sampled. Rotation 
groupr(r = 1, 2, ..., k + c — 1), denoted by G,, consists of m, sample units. On occasion 
t(t = 1, ..., k), the sample consists of the units in G,, G;,,, .-., G;4 -_1, so that the total 
sample size on occasion ¢t, mn, = m,; + m4, + ... + Ms4¢--1- On occasion t + 1, G,; is 
dropped from the sample and G,,,. is added. Each rotation group is chosen without replace- 
ment from previously unchosen units in the population. The total sample size over all k 
occasions is m = n; + ny + ... + my. The maximum number of occasions that a unit 
remains in the sample is c. 

If c is small, then estimates of the correlation parameters in ( can be unstable, leading 
to instability in the estimates of interest (ji;, f2, ..., fA) 7. Viewed another way, the total 
number of parameters is at least k + 2 and increases with time, i.e. with the addition of new 
occasions. Since the dimension of the parameter space increases with time, maximum like- 
lihood estimates of parameters may be biased and inconsistent. The problem of the stability 
of estimates has been addressed in sampling on successive occasions, for example, by Blight 
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and Scott (1973), who assume that the elements of (1), 12, ..-, ux)? follow a time series 
process. On using this assumption the dimension of the parameter space is fixed at a relatively 
small number so that the problems of instability, bias and inconsistency are resolved. In this 
paper, a different approach is taken. Here the fixed means assumption is retained and marginal 
and approximate conditional likelihoods are derived for the parameters in Q, treating the fixed 
means as nuisance parameters. 

Marginal likelihood estimation was introduced as a general method for eliminating nuisance 
parameters from the likelihood function (Fraser 1967; Kalbfleisch and Sprott 1970). Cox and 
Reid (1987) introduced approximate conditional likelihoods which also address this problem. 
They argued that the approximate conditional likelihood was preferable to the profile likelihood 
obtained by replacing the nuisance parameters in the likelihood by their maximum likelihood esti- 
mates when the parameters of interest are given. Bellhouse (1990) established the equivalence of 
marginal and approximate conditional likelihoods for correlation parameters under a normal model. 

Following on the work of Cox and Reid, Cruddas ef a/. (1989) obtained an approximate 
conditional likelihood for the correlation parameter in several short series of autoregressive 
processes of order one with common variance and autocorrelation parameters. Based on a 
simulation study, Cruddas et al. (1989) showed that the estimate based on the approximate 
conditional likelihood had a much smaller bias and better coverage properties of the confidence 
interval than the maximum likelihood estimate from the profile likelihood. The situation 
described by Cruddas et al. (1989) applies directly to sampling on successive occasions in sample 
surveys. In order to reduce the response burden, individuals in a survey are retained in the 
sample for relatively short periods of time. It is expected that the use of marginal and approx- 
imate conditional likelihoods will improve the estimates of correlation parameters and conse- 
quently improve the estimates of the mean for each occasion. 

Within a rotation group, the sample measurements on an individual are usually modelled 
by an autoregressive moving average process (ARMA), i.e. the parameters in 2 are comprised 
of the correlation parameters in the ARMA process. See Binder and Hidiroglou (1988) for a 
review of the application of time series models to sampling on successive occasions. Conse- 
quently, it is of interest to obtain marginal and approximate conditional likelihoods under 
ARMA models with application to rotation sampling. The marginal and approximate condi- 
tional likelihoods for the correlation parameters in a normal model are obtained in Section 2. 
The general results of Section 2 are illustrated in Section 3 by applying the results to sampling 
on successive occasions assuming simple random sampling of units in rotation groups. In 
Section 4, some methods are given to apply these likelihood methods to complex surveys. 


2. MARGINAL AND APPROXIMATE CONDITIONAL LIKELIHOODS 
FOR CORRELATION PARAMETERS UNDER 
A NORMAL MODEL 


Let y be a vector of sampled observations of dimension m x 1 which follows the linear model 
y =XB+€ (2) 


with error vector € ~ N(0,07@), where @ is the m x mcorrelation matrix and where 8 is the 
p X 1 vector of regression coefficients so that X ism x p. The log-likelihood for 8, 0? and 
® is given by 


L(6,o7,®) = — {mIno + (In| | )/2 + (y — XB)76—"(y — XB)/(207)}. GB) 
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For a given value of ®, 
B = (X7871X)X7@~ly 
and 
s? = yT@-y — yT@-1X (X7O-1X)-! XTO“!y (4) 


are jointly sufficient for 8 and o7. 


A marginal likelihood for ® is obtained by making a transformation of the data y to the 
sufficient statistics 8 and s? and the ancillary statistic 


Gib eay XX INNS) XT holy) /s, 


where ®~” is the m x m dimensional matrix such that @=! = @~“%@~”. The marginal 
likelihood for ® is the marginal distribution of the ancillary a times the product of the differ- 
entials da;,i = 1, ..., m. See Kalbfleisch and Sprott (1970, eqs. 6 and 10) for a general 
discussion and a general expression for IIda;. Bellhouse (1978) and, later independently 
Tunnicliffe Wilson (1989), showed that the marginal likelihood for ® under the normal model 
is given by 


Ly(&) = {| 61% | XOX | sR. (5) 


Note that (4) is proportional to the maximum likelihood estimate of o” given ® and that 
s?(X7~!X) —' is proportional to the estimated variance-covariance matrix of the maximum 
likelihood estimate of 8B given ®. Then (5) can be written as 


Ry |% 
i ey Se ND le (6) 
suit oi 12 


To obtain an approximate conditional likelihood, it is first necessary to transform the 
parameters to achieve parameter orthogonality between the parameters of interest and the 
nuisance parameters, which now may depend on the parameters of interest. Sets of parameters 
are orthogonal if the associated information matrix is block diagonal, with each block as the 
information matrix for each parameter set. The conditional likelihood is related to the distri- 
bution of the data y conditional on the maximum likelihood estimate of the nuisance parameters 
for fixed values of the parameters of interest. The approximate conditional likelihood is 
obtained by applying two approximations to this conditional distribution. See Cox and Reid 
(1987, Section 4.1) for a discussion of the derivation. For example, let 6 be the vector of 
parameters of interest and let A, possibly depending on 9, be the vector of nuisance parameters 
orthogonal to 9. The full likelihood of the data for parameters 9 and A is denoted by L(0,A) 
and the profile likelihood for @,L(0,A) is the likelihood with A replaced by its maximum 
likelihood estimate. The approximate conditional likelihood for © is 


TNR) |PTCORA) F*; 


where J(0,A) is the observed information matrix for A at a fixed value of @. See Cox and Reid 
(1987, eq. 10). 
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Following Cruddas et a/. (1989), Bellhouse (1990) suggested, for model (2), the parameter trans- 
formation \ = Ino + (In | Q |)/(2m) leaving 6 the same. The log-likelihood under the new 
parameterization is denoted by L (8, \, ®) and can be obtained from (3). If the entries of ® are 
functions of a parameter ¢, then the nuisance parameters ) and @ are each orthogonal to ®, i.e. 


2 
- Le[PHOME)] 9 
m 0g0Xd 


and 


2 
- Le[PEBAH) «0 
m d¢0B 


when each entry of ® is a continuous and differentiable function of ¢. Moreover, in this case 
the approximate conditional likelihood for &, Lc(®) is the same as the marginal likelihood 
La(®), given by (5) or (6). See Bellhouse (1990) for details. 

The marginal and approximate conditional likelihood in (5) or (6) can be evaluated at any 
® using state space models in the approach of Harvey and Phillips (1979). For any given ®, 
once the recursions to estimate 6 and o* are complete, the value of s? and | ® |” can be 
calculated from Harvey and Phillips (1979, eqs. 5.6 and 6.6, and 4.3 respectively). It is then 
necessary only to obtain X7@~!X and its determinant. The value of X7@~!X may be 
obtained from the final step in the recursive equations of Harvey and Phillips (1979, eq. 3.4). 


3. SIMPLE RANDOM SAMPLING ON SUCCESSIVE OCCASIONS 


3.1 Some General Results for Rotation Sampling 


Suppose rotation group G, first appears in the sample on occasion u and last appears on 
occasion v. Then w is either 1 or rand vis either r + c — 1 ork. The total number of occa- 
sions on which a unit in G, is present in the sample is b = v + 1 — u. Let y,,, ..., Vy, be 
the sample means or elementary estimates for G, on occasions u,u + 1,...,v—l,v 
respectively. Then under model (1), the contribution of G, to the log likelihood in (3) is 


— {bn, Ino + (n,/2).In(| Q, |) + [ax7fQ7'x, + (nm, — 1) tr(Q;'S,)]/(207)}, (7) 


where a is the 1 x b vector (Fur — Pur Iutiyr — Putty +++s Iy-1,r — by—1» Ivr = By)» 
where S,is the b x b matrix of sample variances and covariances for observations within the 
rotation group, and where (, is the b x b correlation matrix on the observations on a single 
unit within the rotation group. Note that the parameters in Q as given in expression (1) will 
also be the parameters in (,. The correlation matrix 0 is based on measurements from all 
occasions | through k; the correlation matrix (2, is from the subset of the data observed from 
occasions u through v. By the independence assumption, the full log likelihood is obtained 
by summing (7) over all rotation groups. 

Given the parameters in (2, or equivalently the parameters inQ,, ..., Q,4,_1, expressions 
for the maximum likelihood estimates f and 6, for » and o” respectively, may be found. 
Likewise, V(j.), the estimated variance-covariance matrix of fj may be obtained. This is 
illustrated for a first-order autoregressive process in Section 3.2. Then the marginal likelihood 
for the parameters in Q), ..., Q,4-_, is given by (5) with the expressions in (5) given by 
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k+c-1 


|\@|*= JT] &, 
n= 


Pee Xa = VA aise (8) 
k+c-1 
sc (m7 051%, + (n, — 1) tr(971S,)}, 
r=1 


and p = k, where X, is x, with the y’s in x, replaced by their maximum likelihood estimates. 


3.2 First-Order Autoregressive Processes 


When specific forms of the correlation matrices 0,, ..., ,4-_; are used, some simplifi- 
cations to the general form of the marginal likelihood for correlation parameters, given by 
(5) and (6), may be obtained. For example, assume the first-order autoregressive model 


Vg = hee Oa pe Br), Fey (9) 


where. €,; ~ N(0,o*) fort = 1, ..., Kandj = 1, ..., N, and where the €’s are mutually 
independent. Model (9), essentially Patterson’s (1950) model, is a special case of (1). As in 
Section 3.1, the vector of regression parameters B = (py, ..., Mx) T When the data vector 
y contains the measurements on each unit grouped by all the occasions on which it was sampled, 
as in the rotation sampling description of Section 3.1, the correlation matrix ®, now a function 
of the autoregressive parameter ¢, can be written as a direct sum of matrices, each of which 
are the correlation matrices of a first-order autoregressive process. 

The following notation, similar to Patterson (1950), is used to denote various sample sizes, 
means and sums of squares and cross products (corrected for the appropriate mean) for occasion f: 


ma, = the proportion of units on occasion ¢ that are matched with units from the previous 
occasion (¢t — 1); 
n, = the number of units sampled on occasion f; 
J; = the mean of the units on occasion ¢ that are matched with units from the previous 
occasion (tf — 1); 
y;’ = the mean of the units on occasion ¢ that are unmatched with units from the previous 
occasion (ft — 1); 
JY, = the mean of all the units on occasion ¢; 
xX; = the mean of the units on occasion ¢ that are matched with units from the following 
occasion (¢ + 1); 
Syy; = thesum of squares among units on occasion ¢ which are matched with units from the 
previous occasion (f — 1); 
syy;’ = the sum of squares among units on occasion f which are unmatched with units from 
the previous occasion (¢ — 1); 
sxx; = the sum of squares among units on occasion ¢ which are matched with units from the 
following occasion (¢ + 1); 
syy, = the sum of squares among all the units on occasion f; 
sxy/ = thesum of cross products for measurements on sample units from occasion ¢ matched 
with sample units from (ft — 1). 
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Under the special case of model (9), and after much algebra, it may be shown that (7) summed 
over all rotation groups r, the log-likelihood for the data reduces to 


L(pm1, ++) Hes 07, 6) = — mino + (d/2)In(1 — $7) 
— {A(pu,¢) + B(¢)}/(207), (10) 


where d is the distinct number of units sampled (irrespective of the number of occasions on 
which a unit is sampled) and m is the total sample size (n; + ... + n,). Further in (10), 


A(u,o) = (1 — $7)m (9, — wy)? 


k 
Ve ye [mem (97 — py — O(Kf_1 — wy-1))? + (1 — 2), (1 — 67) 9 — H,)7| 
Fee (11) 


and 
k 

B(d) = (1 — $°) sy + YY [b? sxxi_1 — 26 sxy/ + syyé + (1 — 6°) sy}. (12) 
t=2 


For any given value of @ the maximum likelihood estimator is s = G~'z and 6” = 
{A(fi,¢) + B(d)}/m, where A (i,¢) is (11) with p replaced with its maximum likelihood 
estimate and where Gis asymmetric k x k band matrix of band width 3 andzisak x 1 vector. 
The nonzero entries of G are 


Su = ™mN; AF (1 cane ™)n,(1 a *) ae vara ieee fOtese— gs os ey ik 


and 
&t,t+1 — 4111419 for = Ls an k- KF, 


where 7, = %,%4; = 0. The entries of z are 


Ry = BN: — OF-1) + (1 — ayn; 1 — *) — Ky i Ney Vr iGPea) 


fort = 1, ..., k, where 7; = m4, = Oand jy’ = j,. The vector of estimated means {i is 
unbiased for » under model (9) and its variance-covariance matrix is o°G ~'. It follows from 
(5) or (6) that the marginal and approximate conditional likelihood for ¢ is 


(1 we 2) 4/2 


Ly(¢) = ———_—__—___—__. 
{A(f,o) + B(d)}"-? | G |” 


(13) 


3.3 Example 


The data for this example are forestry data taken from Cunia and Chevrou (1969, p. 220). 
The data are the merchantable volume of timber per plot measured on three occasions with 
partial replacement of the sample units. In rotation sampling it is assumed that once a unit 
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Figure 1. Marginal Likehood for the AR(1) Parameter 


is dropped from the sample it is not selected again. In view of this assumption an adjustment 
to the data in Cunia and Chevrou was made. In particular, the measurements from sample 
units matched on the first and third occasions without matching units on the second occasion 
were dropped from the current example. From the remaining data the following calculations 
may be made: 


Map —2S0/ 1395) ey = 76/100,.) ny) = 104,007) = 139 13% = 5100, oo P3=9161.5581, 
Jz = 179.9211, jy = 154.0673, jy = 167.2075, ° yy = 181.125, xX; = 147.6512, 
X; = 163.4342, syyz = 864129.2, syy3 = 555369.5, syyj’ = 943948.5, syy = 
266820.7, syy3’ = 271762.6, sxxj = 800753.5, sxx; = 559850.7, sxyz = 812435.7, 
sxy3 = 550943.6, d= 181, and m — k = 340. 


On substituting these data into (13) the marginal and approximate conditional likelihood 
of the data for the autoregressive order one parameter ¢ may be obtained. This is shown in 
Figure 1. 


4. COMPLEX SURVEYS 


There are several ways in which one may proceed to analyze time series data from complex 
surveys. Each method that can be put forward will depend upon the sample information that 
is available. 
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If data are available at the micro level, then variance-covariance matrices based on the 
complex design can be computed for the elementary estimates for each rotation group. A pseudo 
marginal likelihood is obtained by replacing £, and S, in (5) and (8) by their complex survey 
counterparts. A similar approach is taken, for example, by Roberts, Rao and Kumar (1987) 
in logistic regression analysis for complex surveys: obtain a likelihood or a set of likelihood 
equations and replace the usual statistics by their complex survey counterparts. 

Under simple random sampling, S, estimates the finite population variance-covariance 
matrix for measurements on the occasions covered by rotation group 7. Consequently, in a 
complex design, S, is replaced by a design-consistent estimate of the corresponding finite 
population variance-covariance matrix. For example, Kilpatrick (1981) looked at a stratified 
sampling design on two occasions for evaluation of the standing volume of state forests in 
Northern Ireland; the strata were based on the times, beginning in the 1920’s, at which the 
forests were planted. In order to calculate the stratified sampling equivalent to S,, it is 
necessary to have the estimates of the means on each occasion, strata means, strata variances, 
and strata covariances for the unmatched and matched samples from the two occasions. For 
a stratified population, the finite population variance (or covariance) may be decomposed into 
terms comprising the variation (or covariation) between strata and the variation (or covariation) 
within strata; see, for example, Cochran (1977, eq. 5.32). Estimates of the means and strata 
means would be used to obtain a consistent estimates of the between strata variation or covaria- 
tion component and estimates of the strata variances and covariances would be used to obtain 
estimates of the within strata variation and covariation. Unfortunately, only certain strata 
variance and covariance estimates were relevant to Kilpatrick’s study, so that there is insuffi- 
cient published data in the article to calculate a maximum marginal likelihood estimate for 
the correlation between timber volumes on the two occasions. 

In many cases the data at the micro level will not be available. The estimation procedure 
then depends upon the data that are available. One scenario is considered here; others could 
be formulated. Suppose that only the elementary estimates and their design effects are available. 
Let y,;, be the estimate from rotation group G, on occasion ¢f based on a sample of size m,. 
Let deff,,, be the design effect associated with jy, ,. If o*/m, is the variance of ¥;,, under simple 
random sampling, then on appealing to the Central Limit Theorem, 


(Fir — w1)/(deff;,,) * ~ N(0,07/m,) (14) 


approximately. The modelling may proceed by assuming, within G,, an ARMA-type process 
such as 


(9, —. Hy) / (deff, )* =.b(Pe1, — pena) / (deft, eee (15) 


where €, has constant variance. This may be easily cast into the framework of model (2), where 
the data vector y contains data of the form J,,/(deff,,) *, where B is (uy, u2, ---, Hx)”, and 
where X contains entries of the form 1/(deff,_,,-) “. The marginal likelihood, obtained as a 
special case of (5) or (6), may be evaluated using the state space models of Harvey and Phillips 
(1979) as noted in Section 2. Marginal and approximate conditional likelihood estimation is 
especially desirable under the model given by (14) and (15). The estimate of ¢ in this case is 
based on the variation between elementary estimates within each rotation group; the varia- 
tion within elementary estimates is not available. The length of time a rotation group remains 
in the sample is short so that the problems of bias and inconsistency in the maximum likelihood 
estimates will be applicable here. 
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5. DISCUSSION 


Binder and Dick (1990) have also suggested the use of marginal likelihood estimation 
techniques for sampling on successive occasions. In their framework, suppose that the survey 
estimates of the means, say f,, are available for each occasion t = 1, ..., k. Also, the matrix, 
say S, of variances and covariances of the surveys estimates is available. As in Binder and Dick 
(1989, 1990), among several others, the ¥,’s may be modelled by 


Vi = Br + Ce}, (16) 


where é,; is the survey error at time ¢ with variance-covariance matrix estimated by §. The 
means on each occasion, py; for occasion t, follow an ARMA process. Model (16) is a special 
case of the random coefficients regression model, so that the appropriate marginal likelihood 
is different from (5). 

A marginal or approximate conditional likelihood for correlation parameters in a random 
coefficients regression model is obtained as follows. Suppose in model (2) that 8 is a random 
vector modelled by 6 = Wé + u, where W is a p X q matrix of known values, 6 is a 
q X 1 vector of parameters, and u ~ N(0, y7I), independent of €. Under the composite 
model y = XW5 + Xu + €, the log-likelihood for 6, 0,T, y?, and x = 07/7", denoted by 
L(6,x,y*,T',Q), is given by (3), with Q replaced by x9 + XIT'X7 and XB replaced by XW6. 
Likewise, the marginal likelihood, denoted by Ly,(x,T’,Q), is given by (5), with _X replaced by 
XW and © replaced by kQ + XTX’. This yields 


Ly(x,T,Q) = {| «@ +. XTXT |? | (XW)T(KO + XTX") IXW |? g™-B-!, (17) 


where 
yt (RO PONTE Poy 


— yp! (x + XTX!) IXWO(XW) (ko + XTX OCIXW) ACW) (xO XEX?) ay. 


Now the dimension of 2 may be large in comparison to I’; this can be the case in sampling on 
successive occasions. As an alternate approach, one could take the likelihood implied by (3), 
multiply it by the distribution for 6, and integrate over 6 to obtain the likelihood for the para- 
meters under the random coefficient model. This will yield matrices of the same dimension as I’. 

Since §$ is available, an estimate of 0, the correlation matrix of the survey error, may be 
easily obtained. An estimate of x = o*/y”, may also be obtained. From assumptions which 
lead to the marginal likelihood in (17), it is necessary to assume that e, in (16) is a stationary 
random variable. Then an estimate of o? is the average of the diagonal elements in S. If y7 
is the variance of the »’s then the variation between j,, ¢ = 1, ..., k provides an estimate 
of o? + y*. From these two estimates, an estimate of x may be obtained. Under model (16), 
Xin(17)isthek x kidentity matrix, while Wisak x 1 column vector of 1’s. The resulting 
marginal likelihood is a pseudo likelihood since some of the parameters have been replaced 
by estimates. In this case, the pseudo marginal likelihood for the parameters in I’ (pseudo since 
x and Q have been replaced by their estimates) and is given by (17) with the appropriate substitu- 
tions. The parameters in I are the correlation parameters in the ARMA process on yp;. If k, 
the number of occasions, is relatively large in comparison to the number of parameters in I’, 
then the marginal and approximate conditional likelihood estimates should be similar to the 
maximum likelihood estimator. For ease of computation, it seems that the full likelihood 
approach using the state space models as outlined by Binder and Dick (1989a, Section 3) 
appears to be the simplest approach to use in this situation. 
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Sample Rotation and Estimation in the Survey of 
Employment, Payroll and Hours 


IOANA SCHIOPU-KRATINA and K.P. SRINATH! 


ABSTRACT 


The current Survey of Employment, Payroll and Hours, conducted by the Labour Division of Statistics 
Canada is a major monthly survey collecting data from a large sample of business establishments. This 
paper describes the methodology of the survey. The description of the stratification, sample size deter- 
mination and allocation procedures is brief, whereas the description of the rotation procedure is more 
detailed because of its complexity. Some of the possible simplifications of the design are also highlighted. 


KEY WORDS: Establishment; Response burden; Sampling frame. 


1. INTRODUCTION 


1.0 Objectives of the Survey 


The Survey of Employment, Payroll and Hours (SEPH) is a monthly establishment based 
survey conducted by Statistics Canada. 

The main objectives of SEPH are: 

(i) to provide monthly estimates of the total number of paid employees, average weekly 
earnings, average hourly earnings, average weekly hours and other related variables at 
the industry division-province level. 

(ii) to provide the above estimates for Canada at the three digit Standard Industrial Classi- 
fication (SIC) level. 

(iii) to provide standard errors of all the estimates produced. 

It is also intended to produce estimates at the three digit SIC-province level annually. 

The survey covers all industries with the exception of agriculture, fishing and trapping, 
private household services, religious organizations and military services. For a detailed descrip- 
tion of the objectives and uses of SEPH, see Cottrel-Boyd et al. (1980). 

This article describes the sample selection and rotation as well as the estimation procedure 
adopted for the survey. Chapter 2 presents the sample selection and rotation procedure in detail. 
Chapter 3 is devoted to the estimation procedure. Some of the details relating to Chapter 2 
are given in the Appendix. The Appendix also presents a simplified estimator of the number 
of live units. 

For a complete description of the SEPH methodology, see Schiopu-Kratina and Srinath 
(1986). 


1.1 Preliminary Definitions 


Some of the terms used in this article are defined here for convenience. 


(i) Establishment - An establishment is the smallest unit that is a separate operating entity 
capable of reporting all elements of basic industrial statistics. The establishment is the 
statistical unit for SEPH. We will use the term unit for establishment. 


' Ioana Schiopu-Kratina and K.P. Srinath, Business Survey Methods Division, Statistics Canada, Ottawa, K1A OT6. 
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(ii) Employment reporting unit (ERU) - For purposes of detailed geographical statistics, the 
establishment is often sub-divided into reporting units based mainly on location and 
sometimes on other considerations like payroll, efc. 

(iii) Standard Industrial Classification (SIC) - 1970. Each establishment is assigned a Standard 
Industrial Classification (SIC) code according to the nature of its activity. These SIC codes 
are defined in the SIC Manual (Statistics Canada 1970). For the purpose of the survey, 16 
industry divisions (groups) which are groupings of specific three digits SIC’s have been created. 

In SEPH, the total number of paid employees associated with a unit is the characteristic 
chosen as a measure of size of that unit. 

There are four size groups in SEPH and their boundaries are defined as follows: 0-19 for 
size group 1, 20-49 for size group 2, 50-199 for size group 3 and 200 or more for size group 4. 
(iv) A super-stratum is defined by an industry division, province and size group. With 16 

industry divisions, 12 provinces and territories and 4 size groups, there are 768 super-strata. 

(v) A stratum is defined as a three digit SIC, province and size group. It is the finest level 
of detail for which estimates are obtained. 

(vi) The take-all portion of the population consists of units which are all included in the sample 
with certainty. It contains units in size group 4 and pre-specified units of the population. 
The take-some portion of the population consists of the remaining units which are subject 
to sampling as described in the following sections. 


2. SAMPLE SELECTION AND ROTATION 


2.0 Sample Size Determination and Allocation Procedure 


In SEPH, the take-some sample size is determined at the industry group - province level 
based on a designed coefficient of variation of the estimate of the total number of employees for 
that industry group-province. The required sample size and the sampling fractions are calculated 
at the super-stratum level, using _X-proportional allocation, where X represents the total number 
of employees. The sampling fractions are held constant from one month to the next. Details 
about the allocation procedure can be found in Schiopu-Kratina and Srinath (1986). 

The actual selection is made at the stratum level. Due to the minimum sample requirements 
at that level, the number of sampled units is larger than the required sample size at the industry 
group - province level (see (2.2)). 


2.1 Sample Selection 


Let us now consider a specific stratum. Let N be the size of the take-some portion of the 
population and n the size of the take-some sample in this stratum. 

Whereas the allocation of the sample to the super-strata is X-proportional, the allocation 
at the stratum level within each super-stratum is essentially proportional to the total number 
of units in the take-some portion of that stratum. 

The sampling fraction in each stratum is given by the formula: 


1 
= max( f’,— _}, wel 
femiags| an 
where f’ is calculated at the corresponding super-stratum level. In order to reduce the instability 
of the estimates caused by small values of the sampling fraction, it was decided to set 1/100 
as a minimum sampling fraction for all strata. 


Survey Methodology, June 1991 81 


The detailed calculations of the sample size at stratum level are given in section 2.3 (see the 
derivation of the formula (2.8)). A systematic sample is drawn from each stratum. 


2.2 The Rotation Scheme 


The sample rotation (partial periodic replacement of the sample) in SEPH is designed 
primarily to reduce the response burden. From previous surveys, it appeared that the average 
response rate in strata in which there was no rotation was significantly lower than the average 
response rate in strata in which there was rotation. Also, the existence of a large portion of 
units common to the sample for two consecutive months improves the reliability of the esti- 
mates of month-to-month change relative to the estimate of change based on two independent 
monthly samples. Rotation of the sample in each stratum has to be done under certain 
constraints such as keeping the units out of the sample for a certain period of time after they 
rotate out of sample. 

The monthly sample consists of 14 groups numbered from 0 to 13. Group 0 contains the 
take-all units of the stratum. Groups | to 13 are called rotation groups. The labels 1 to 12 on 
rotation groups indicate the month in which the units other than births rotated into the sample. 
For example, rotation group | contains mostly units which entered the sample in January and 
births, rotation group 2 contains mostly units which entered the sample in February and births, 
etc. Rotation group 13 contains units which have completed 12 months in the sample. These 
units are the oldest in terms of the time spent in the sample and are eligible to be rotated out. 
Each month, births are selected and allocated at random to the rotation groups. 

At the time of the monthly selection and rotation, all units in the reference month are 
transferred to rotation group 13. In February, for example, all units in the rotation group 2 are 
transferred to rotation group 13. A replacement group is selected from ‘‘eligible for selection’”’ 
units, and newly recorded units (births). The units of the replacement group (with the exception 
of 11/12 of all births) are then placed in rotation group 2, and they are not eligible to rotate 
out for at least 12 months. If sufficient units are available for a replacement group, the contents 
of group 13 are removed from the sample and are not eligible for reselection for 12 months. 
Otherwise, some units in group 13 are retained in the sample until such time that there are 
enough available units outside the sample to form a replacement group. This is done in order 
to maintain the minimum sample size or attain a sample size large enough to provide estimates 
with prespecified reliability. This way, in general at least 11/12 of the units stay in the take- 
some portion of the sample for two consecutive months. 

The units that have left the sample are assigned to a waiting group which is divided into 
subgroups. A subgroup consists of units which were all removed from the sample in the same 
month. The waiting group contains 12 subgroups in every stratum. The time each unit has spent 
outside the sample is thus recorded to ensure that the units will not be reselected for at least 
12 months. The units that have spent the required amount of time in the ‘‘not eligible for 
selection’’ group are transferred to the ‘‘eligible for selection’’ group and are thus assigned 
a positive probability of reselection. 


To summarize, the entire take-some population at any given time consists of four groups 
of units. These are: 
(i) units that are in the sample for that month; 
(ii) units that are eligible for selection (E.F.S.); 


(iii) units in the waiting group which have rotated out of the sample less than 12 months ago 
and which are not eligible for selection (N.E.F.S.); 


(iv) births, 7.e. units that have not been previously recorded on the frame. 
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The process of monthly selection and rotation involves an exchange of units among these 
groups. Some units leave group (i) for group (iii) and new units enter group (i) from group 
(ii), after some selected births from group (iv) have been transferred to group (ii). The remainder 
of the births are allocated to groups (ii) and (iii) after selection. This is done in order to insure 
that the sample is representative of the population in any given month. 


2.3. Determination of the Sample Size and Weights 


2.3.1 Monthly Updates 


The sampling frame contains a large number of units which are inactive, out of business, 
out of scope etc. Apart from the burden of retaining an increasing number of inactive units 
on the frame, the estimators based on samples drawn from such a population are likely to have 
a high variance, due to the fact that the sample contains a high proportion of zero observations. 
Ideally, all such units should be eliminated from the sampling frame before the monthly sample 
is drawn. The frame is updated each month, after a monthly selection and rotation and prior 
to the next. For this reason, the indices we use to denote births and deaths on the frame are 
one unit higher than those used for the sample size in the sample selection preceeding the update. 
For example, after the initial sample selection, say (0) units are in the sample, of which d(1) 
units are subsequently found to be dead units. Then D(1) denotes the number of dead units 
in the out-of-sample portion of the population and B(1) the number of units registered as births 
that month. In calculating the required sample size for the following month n(1), one must 
take into account these updates (see (2.3)) as well as the size of the population at the time of 
the first sample selection N(0). 


2.3.2 Determination of Sample Size 


The population of a given take-some stratum is a function of time and it will be denoted 
by N(t) say, where ¢ is a positive integer which increases by one unit from one month to the 
next. The required sample size is, for each month: 


n’(t) = [fN(t) + 0.5]. (2:2) 


Here [a] is the largest integer number which is not greater than a. The constant 0.5 is used 
for a better approximation in the rounding off procedure. 

Suppose d(f) units are eliminated from the sample (in-sample deaths) and D(t) from the 
rest of the population of the stratum (out-of-sample deaths). Also let B(t) new units be recorded 
during the same time interval (births). 


As a result, the size of the population of the cell at the time of the ¢'" selection is: 
N(t) = N(t — 1) — d(t) — D(t) + Bt). (2.4) 


Since the updates are not exhaustive, undetected inactive (dead) units are expected to exist 
in the population. 


Let n,(t) be the number of live units left in the previous month sample (after the updates) 
L.e.: 


nt) = n(tie 1)o-— d(t). (2.4) 
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We assume that there are no undetected dead units in the sample at this point. We can 
think of the population of a stratum as consisting of two domains: the domain of live units 
and the domain of dead units. The size of the dead domain is not known, but an estimate 
U,,(t) can be calculated based on the information given by the sample and the updates (see 
Appendix). Let U,(t) be an estimate of the number of undetected dead units in the population 
at the time of the ¢th monthly selection. Then: 


N(t) = U,(t) + U,(t), (2.5) 


where U,(t) is the estimate of the number of live units. 


The probability of choosing a dead unit when selecting a unit at random from the out-of- 
sample units is: 


Z Uq(t) 
P,(t) = min} ——————., : (2.6) 
N(t) —"n,(7) 
The required number of live units in the sample is: 
né(t) = fU/(t). (2.7) 


The replacement sample size is calculated in such a way as to ensure that the expected number 
of live units in the sample after selection is 7; (ft). 

Now assume that at the time of the ¢t8 sample selection and rotation, of the 7,(f) live units 
in the sample, n,(¢) units are eligible to rotate out. 

Since there are n,(t) — n,(f) live units left in the sample, n/(t) — n,(t) + n,(t) more live 
units are required in the sample for the ¢t4 month. 

In order to represent the births in the sample adequately, b(t) = /B(t) births should be 
selected at random and included in the sample. 


Therefore: 
&(t) = max(n;(t) — n(t) + no(t) — b(t), 9), 
live units should be selected from the eligible for selection group and added to the sample along 


with the selected births. Taking into account the existence of an unknown number of inactive 
units in the population and integerizing, it is required that: 


0) 


more units rotate into the same sample, with P,(t) given by (2.6) andn” (t) = N(t) — n,(t). 


f(t) 


—#§_—— + 0.5 
1 ors) 


nt) = min ( 


In calculating n;(t), we made the assumption that there are no inactive units among the 
births, so the expansion factor [1 — P,(t)]~! is applied only to the ‘‘older’’ units in the 
E.F.S. group. 


The sample size n(t) for the ¢t® month is: 


n(t) = max{n,(t) — n,(t) + n,(t) + b(t), m}. (2.8) 
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In (2.8), m represents the minimum required sample size for a stratum, which is presently 
set at 3. This additional requirement increases the sample size by 3,000 units in all strata, of 
which 1,800 are expected to be in the sample for a considerable length of time. 

Of the n;(t) units which rotate in, Ag(t) = P,(t)n;(t) are expected to be found inactive 
and 7i,(t) = n;(t) — Ag(t) active. Thus, of the n(f) units in the sample after the tth monthly 
selection and rotation, #,(t) = n,(t) — no(t) + A,(t) + b(t) are expected to be alive and 
they represent the U,(t) units of the live domain at the proper rate f (see (2.7) - (2.8)), when 
n(t) > min (2.8). 


2.3.3 Determination of Weights 


The weight w(f) used for estimation for the ¢'4 month is expressed in terms of the size of 
the population and sample at time ¢. However, the use of N(t)/n(t) as weight for estimation 
could lead to an overestimation of the live units in the population. Indeed, n(t) in formula 
(2.8) was chosen so that the expected number of live units in the sample equals the required 
sample size. The number of dead units in the sample drawn as described above may not repre- 
sent the size of the dead domain at the proper rate. In (2.8), n; is drawn from the general 
population and is thus expected to preserve the proportion between the dead and the live 
domain. No deaths are expected to be found among births and thus b(t) properly represents 
the birth subgroup of the population. There are, however, n,(t) — n,(t) units left in the 
sample from a previous selection, after rotation and the updates on the sample. The propor- 
tion of deaths among them is likely to be much smaller than the corresponding proportion in 
the general population, in spite of the fact that the updates are based on information from 
sources other than the survey. Then the value of N(t)/n(t) should be adjusted for the under- 
representation of dead units in the sample. This gives, when n(t) > m, 


igi deegne aly 


toe —"5n (2.9) 
WER yes ee 
where &(t) will be determined subsequently (see (2.10)). The value of #(¢) represents the 
‘*deaths’’ that have to be added to the sample to correctly represent the dead units in the popula- 
tion. Notice that when the first sample is drawn or if a redraw takes place, such an adjustment 
is not needed, i.e. 7(0) = 0. 

In order to find a formula for @(f), we use (2.5) in the numerator of (2.9) and (2.8) in the 
denominator. By (2.7) - (2.8), we must also have: 

U4 (t) 1 


Ai uit 


With A4(t) = P,(t)n;(t), we obtain from above 
e 1 A 
Uj(t) = al) + a(t)] or a(t) = fUg(t) — Ag(t). (2.10) 


The death adjustment is given by: 


a(t) if a(t) = — Ag(t) 
v(t) = (2.11) 
0) iwi WEL) <n gta) 
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and the weight used in estimation is: 


we N(t) 


~ n(t) + 9(t) ee 


Note that the weight in (2.12) is defined using an estimate and so it is a random variable. 


The use of the weight defined by (2.12) implies that the estimate of the number of live units 
in the population, defined by U,(t) = w(t)A,(t) does not exceed N(t), the size of the popula- 
tion at the time of the ¢'4 sample selection. 


Let us define: 


O4(t) = w(t) [0(t) + Ag(t)). 


By (2.10) - (2.11), it follows that U,(t) =O and its minimum value is 0 when 
$(t) — fig(t) = O. By (2.5), the maximum value of U;,(t) is then N(t). 

The restriction that the estimator of live units be truncated at N(t) has implications on 
estimation which will be discussed in section 3.1. The estimator U,(t) is calculated recursively 
(see the Appendix) using 2.10 - 2.11 and the fact that (0) = “#(0) = 0. 

It has to be noted that the formula (2.11) is slightly different from the formula giving the 
death adjustment in SEPH. Firstly, for the sake of simplicity, we did not consider here the 
cases when the minimum sample size m has to be used. In such cases, the use of the sampling 
fraction f in (2.10) is not appropriate. In SEPH, the previous month weight is used in (2.10) 
in lieu of f in all instances. Secondly, the death adjustment in SEPH is always taken to be 
positive. Formula (2.11) shows that it could be negative, as long as it is larger than — A,(f). 
The actual instances in which this happens, or, more generally, when @ < 0 are very rare. 

The Appendix presents a formula for the estimator of the live units which does not require 
the use of the death adjustment. 


2.4 Sampling of Births 


As mentioned previously, every month new units are added to the frame. Since it is believed 
that these new units (births) may differ from the ‘‘old’’ units, a special birth strategy was 
designed, aimed at adequately representing the births in the sample. 

Ideally, if B births are added during the current month and if f is the stratum sampling 
fraction, then b = /B births should be selected in the sample during that month. The selected 
births are randomly assigned to the rotation groups described in section 2.2. This ensures the 
same probability of rotating out of the sample for births as for the ‘‘old’’ units, so that the 
age distribution of the in-sample units is the same as the out-of-sample units. 

Using the notation of the previous section, let n; be the number of units required to rotate 
in at the time of the monthly sample selection excluding births and N’ the number of units 
in the E.F.S. group (group (ii) of section 2.2). The birth strategy consists of a two-phase selec- 
tion procedure. This procedure involves the formation of acommon pool of births and ‘‘older’”’ 
units from which a sample is then drawn. This was thought necessary because usually, the birth 
group is too small for sampling births separately each time. There are two ways of forming 
the common pool depending on the sizes of the birth group and the E.F.S. group. If: 


= (2.13) 
may ws é 
N’ f 

then b’ births are preselected from the birth group and a common pool of size (N’ + b’) 
is formed where: 
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eS es f (2.14) 
nj 


Inequality (2.13) ensures that b’ < B which means that the birth group is large enough and 
the preselection can take place. From the common pool of N’ + b’ units, n; + b units are 
selected next and added to the sample. 

The choice of b’ as given by (2.14) ensures that the expected number of births in the sample 
is the desired one, since the probability of selecting one birth from the pool is b’/ (b’ + N’) 
and therefore the expected number of births when 1; + b units are selected without replace- 
ment from this pool is (n; + b) b’/(b’ + N’) which by (2.14) equals b. Similarly, it is easy 
to see that the expected number of ‘‘older’’ units is 7;. 


In the complementary situation, when (2.13) does not hold, acommon pool of size (n’ + B) 
units is formed where: 


n’ =n,/f <N’. (2.15) 


Then n’ ‘‘older’’ units are selected from the E.F.S. group. Let us note that in this situation 
b’ as given by (2.14) is larger than B and so the first procedure cannot be applied. 

We now calculate the expected number of ‘‘old’’ units in the sample. Since the pro- 
bability of selecting one ‘‘old’”’ unit is now n’/(n’ + B) andn; + bunits are drawn from 
the pool of n’ + B units and placed in the sample, the expected number of ‘‘old’’ units is 
(n; + b) n’/(n’ + B) which equals n;. The expected number of births is b. 

It has to be noticed that an underrepresentation of births may occur in some situations. For 
example, if in some stratum no units are required to rotate in, then for the month in question 
the births will not be represented in the sample taken from that stratum. However this situation 
usually arises when the population size is small and in the long run, the representation of births 
can be expected to average out correctly. 

The b births actually selected are randomly assigned to the rotation groups 1-12 in the sample, 
resulting, on the average, in b/12 births assigned to each of these rotation groups. This ensures 
that the probability of a birth rotating out is the same as that of any other unit. 

In order to keep the age distribution of the units in the groups (i) - (iii) (see section 2.1) 
constant, the non-selected births will be allocated to the E.F.S. and the N.E.F.S. group at 
random. That is, if N’ is the number of units in the E.F.S. group and N” is the number of 
units in the N.E.F.S. group, then N’ (B — b)/(N’ + N”) unselected births will be assigned 
to the E.F.S. group and N” (B — b)/(N’ + N”) to the N.E.F.S. group. 


3. ESTIMATION 


3.0 Introduction 


In this chapter we describe the procedure for estimating the characteristic ‘‘the total number 
of paid employees’’. 

As indicated in section 2.0, only the reliability of the estimates of the total employment at 
the industry group-province level of aggregation is prespecified. The estimates of characteristics 
other than the total employment have varying degrees of reliability. For example, estimates 
of average weekly earnings are expected to have higher reliability than the total employment. 

The finest level of aggregation at which the estimates are published is the SIC-province level, 
but the basic ‘‘building blocks’’ for producing the estimates are the strata. 
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An outlier in SEPH is an observation in the take-some portion of the sample which is larger 
than a prespecified value. 

The weight of each stratum is first calculated as in section 2.3, then it is adjusted for outliers 
in that stratum. Estimates of the total employment are computed for each stratum using the 
adjusted weights. There is no adjustment for nonresponse, as values for the nonrespondents 
are imputed. The estimate of the total employment for each stratum is obtained by adding the 
following totals: 


(i) the total employment for take-all units 
(ii) the total employment for outlier units 
(iii) the sum of the weighted values of employment for take-some units, excluding outliers. 


Since the weight assigned to each outlier is one, outliers are treated as take-all units for the 
purpose of estimation and are therefore not used in the variance estimation. 


3.1 Estimation of the Total of a Characteristic 


Let us consider a specific stratum for a given month of the survey. Let N be the size of the 
take-some portion of the population of the stratum and n the number of take-some units in 
the sample for that month. If ¥ is the death adjustment (see (2.11)), then the original weight 
assigned to each unit in the sample for the purpose of estimation is (see (2.12)): 


N 
w= * (3.1) 
n+y 


However, if ¢ outliers are present in the sample with t = 1 then this weight is modified by 
giving each outlier a weight of 1 and assigning to the remaining units in the take-some portion 
of the sample the weight: 


Neer 
ye (3.2) 
Ney et 


Let S represent the set of in-sample units. If Y(u) represents the value of employment 
corresponding to the unit uw in the sample, then the estimate of the total employment in the 
stratum is: 


Y = 3 w(u)Y(u). (3.3) 


ueS 


Where w(u) = 1 if wis an outlier or a take-all unit and w(u) = w’ for all other units in 
the sample (see (3.2)). 

Estimates of totals at any level of aggregation higher than the stratum are obtained adding 
the estimates of the stratum totals, for all strata in the level of aggregation considered. 

Since for the purpose of estimation the outliers may be considered take-all units, we may 
replace, for the sake of simplicity w’ by w, with w given by (3.1). Then N and 7 should be 
modified accordingly (see (3.2)). 

Let N,; be the size of the live domain in the population of the stratum and n, the number 
of live units in the sample. Let Y; represent the average employment of the live units in the 
sample. 
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Since only the live units contribute to the total employment, an estimate of the cell total is: 
¥; = UO en (3.4) 


We consider the case fN > m. 


In (3.4), U/ is an estimate of N,, the number of live units in the population and is given by: 


of veiect 
Uf = ras (3.5) 


In (3.5), fis the stratum sampling fraction which is held fixed. The estimator based on the 
values of U/ is unbiased, that is: 


E(U;): = Nj. (3.6) 


As a consequence of the unbiasedness of the estimator based on (3.5), U/ may exceed N 
in some instances, as low possible outcomes of U; compensate for it. 


In SEPH, the estimate of live units U, is defined by: 
U, = WNp (3.7) 


with the weight given by (3.1). 
The definition of the weight in SEPH (see the end of section 2.3) implies that: 


GjxhifsenadsalN 
Opi (3.8) 
IN Stitt AEN. 


The estimate of the total employment in SEPH is defined by: 


ve = of Y,. (3.9) 


An estimate of the total employment based on (3.5) is: 
Yin= Ue. (3.10) 


The estimator based on (3.8) is biased and consequently the estimator of the total employ- 
ment in SEPH is also biased. 

However, the mean square error of the estimator based on (3.9) conditioned on 7, is smaller 
than the mean square error of the estimator based on (3.10), conditioned on n,. We now 
sketch a proof of this claim. 

It is not difficult to see that, for each particular outcome, the bias B, of the estimator based 
on (3.9) and conditioned on 7 is given by: 


Be= (U, — Ni) ¥p. ith) 
Similarly, we obtain for the estimator based on (3.10): 


Bi = (UO; -s N,) ¥). (3.12) 
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We show that the conditional mean square error of the estimator (3.9) is smaller than the 
conditional mean square error of the estimator (3.10). The same result then holds for the uncon- 
ditional mean square errors. We condition on the realized sample size of live units ny. From 
(3.8) - (3.10), Var[U; ¥; | nm] — Var(O,¥,| m] = [nz f-? — N7]1{m > fN}Varl¥;, | nm]. 
Notice that n?f~? — N* > 0on the set {n, > fN}. We now compare [B;/ ]? and B?: 


[By]? — BP = 1{mf-! > N}{(Of — Np]*¥? — (N — M)*¥?}. 


But Uf — N, =af 27, = Ni >N- N; On ee > N. 

Therefore [B/]” — B? = 0. Since MSE [U/ ¥, | nm] = Var[U; ¥; | nm] + [Bj ]* and 
MSE [U,Y, | m] = Var[U,Y, | n,] + [B,]*, the term-by-term comparison leads to the con- 
clusion that: 


MSE[U/ Y, | m] = MSE[U,Y, | n]. 


This important property motivates the choice of the estimator (3.9) over (3.10) for the total 
employment in SEPH. 


APPENDIX 


In this Appendix, we use the notation of section 2.3.2. 


We first derive the formula for U,(t), the size of the dead domain used in SEPH for the 
th selection. 

Recall U,(t) = w(t) [¥(t) + fAg(t)]. At the time of the tt update, d(t + 1) dead units 
are found in the sample. We can replace therefore /,(t), the estimated number of dead units 
in the sample by d(t + 1) in order to obtain an update of U,(t), namely Nj(t + 1): 


N,(t + 1) = w(t) [0(t) + d(t + 1)). ahes) 


Formula (1.1) uses the death adjustment from the previous month. The initial value of ¥ 
is §(0) = 0 (see the remark after (2.9) and the definition of #7) and w(0) = f~!. Now the 
estimate of the size of the dead domain for the (¢ + 1)t® monthly selection is: 


U,(t + 1) = max(N,(¢t + 1) — D(t + 1) — d(t + 1), 0). (1.2) 


Notice that U,(t + 1) can be calculated from (2.5) when U,(t + 1) is known and vice 
versa. An alternative form for U;(t + 1) is obtained recursively as follows. Let us assume 
that U,(t) is known before the t + 1" selection of th sample (recall that t = 0 is used for the 
first, or original selection). Then U,(t) is also known and can be used to calculate P,(t), the 
probability of selecting a dead unit from the out-of-sample units (see formula 2.6). This 
probability is then used to calculate the required number of units which should rotate in as 
described in 2.3.2, as well as the expected number of live units in the sample at the time of 
the (t + 1)" selection, i,(t). 

Then the weight used in estimation for the next selection is U,(t)//,. After selection, the 
(t + 1)th update takes place and the actual number of live units in the sample is found to be 
no(t + 1). The estimate of the size of the live domain for the following selection can be 
calculated 
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U,(t) 
D n(t+1)+ B(it+1),NGt+ | 


U,(t + 1). = min 
e 


and so forth. To initiate the process, note that the weight used in the first estimation is 
w(0) = f~' and after the first update, 


O,(1) = min{w(0) x m(1) + BU), N()}. 
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Estimating a System of Linear Equations 
with Survey Data 


PHILLIP S. KOTT! 


ABSTRACT 


This paper develops a framework for estimating a system of linear equations with survey data. Pure design- 
based sample survey theory makes little sense in this context, but some of the techniques developed under 
this theory can be incorporated into robust model-based estimation strategies. Variance estimators with 
the form of the single equation ‘‘linearization’’ estimator are nearly unbiased under many complex error 
structures. Moreover, the inclusion of sampling weights in regression estimation can protect against the 
possibility of missing regressors. In some situations, however, the existence of missing regressors can 
make the estimation of a system of equations ambiguous. 


KEY WORDS: Sampling weights; Putative missing regressor; Robust; Nearly unbiased. 


1. INTRODUCTION 


Kott (1991) showed that design-based techniques developed for estimating a single linear 
regression equation could be exploited in a more conventional model-based framework. In 
particular the use of sample weighted regression was shown to help protect against the possible 
existence of missing regressors, while the so-called linearization variance estimator was shown 
to produce nearly unbiased estimators of mean squared error for many complex variance 
structures. 

This paper extends those results to the estimation of a system or ‘‘grouping”’ of linear 
equations, a topic of considerable interest to econometricians (see, for example, Johnston 1972, 
pp. 238-241). Two simple examples may shed some light onto the subject for those not already 
schooled in econometric methods or their equivalent. 

Suppose we have a sample of farmers and want to estimate the relationship between the 
amount of planted soybean acres and the size of the farm. Zellner (1962) showed in effect that 
even if a simple quadratic equation with independent and identically distributed errors correctly 
described the universe, a better estimator than the one produced by ordinary least squares 
(OLS) might exist. This estimator could be found by taking into consideration other linear 
relationships, say between planted corn acres and farm size, that had errors terms correlated 
with those in the original relationship. Zellner called the system-wide estimation of a group 
of such equations ‘‘seemingly unrelated regression.’’ Oddly, in order for Zellner’s generalized 
least squares (GLS) estimator to produce different results from OLS, it is necessary for some 
equations to contain regressors not found in other equations. Alternatively, one can think 
of each equation as containing the same regressors but with certain coefficients constrained 
to zero. 

A second example concerns a sample of firms each producing one output, y, from two 
inputs, x; and x, with unit prices, p, and p>. Economists often assume that each firm 
possesses the same technology (plus or minus an error term). Given pj, p>, and y, each firm 
would choose x, and x, so as to minimize total cost, c = p,xX,; + P2X2. Suppose that the 
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relation between p), P>, y and the cost minimizing c can be expressed by the following equa- 
tion (on average): 


log(c) = bo + b,log(p;) + b2log(p2) + b3log(y). (1) 


Economic theory tells us that a rational firm faced with implicit cost equation (1) would 
choose its level of x; so that 


X;pi/c = by. (2) 


Naturally, in order to estimate equations (1) and (2), we need to add a stochastic structure. 
For simplicity, assume that both equations (1) and (2) fit the behavior of all firms subject to 
respective independent (across firms) and identically distributed random errors. Observe that 
in addition to the strong possibility that the error terms in the two equation will be correlated 
for a particular firm, there is also a coefficient (b,) shared by both equations. 

When faced with a system of linear equations in which the coefficients are known to be 
constrained, the design-based approaches to linear regression reviewed in Kott (1990a) make 
little sense. For that reason, although design-based practice inspires many of the procedures 
discussed here, only the extended model-based approach introduced in Kott (1991) will be used 
to justify them. 

Section 2 lays out the theoretical model for the estimation of a system of linear equations 
based on data from the full population. Section 3 introduces the sample weighted analogues 
of full population OLS and GLS estimators for a system of linear equations. Section 4 addresses 
robust mean squared error estimation of both the sample weighted OLS and GLS estimators 
employing a straigtforward generalization of the linearization variance estimator (see, for 
example, Shah, Holt, and Folsom 1977). Section 5 discusses a general method for developing 
test statistics that can be used to evaluate, among other things, whether sample weighted 
OLS and GLS are actually estimating the same thing. Section 6 explores a simple example. 
Section 7 sketches an extension of the methodology developed here to what econometricians 
call ‘‘simultaneous equations.’’ In the stochastic version of equation (1), for example, many 
economists believe that log(y) should be treated as a random variable and that log(c) can be 
assumed to be fixed. This causes a simultaneity bias if not specifically addressed by techniques 
like two and three stage least squares (see Johnston 1972, pp. 341-420). Finally, section 8 
contains a brief discussion. 


2. FULL POPULATION ESTIMATION 


2.1 The Unconstrained System: 


Suppose we have a population containing M data points. Each data point / is associated 
with G + K observed variables satisfying the following model: 


Y=XB+U+ VJ, (3) 


where Y isanM x G matrix of observed dependent variables (the jth row of Y contains the 
dependent variables associated with the /th data point), 


X isanM x K matrix of observed independent or regressor variables (the ith row of 
X contains the independent variables associated with the /th data point), 
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isan K X G matrix of parameters, 


Ri 


U isanM x Gmatrix satisfying the relationship limyy_..X’U/M = Og,.c (a matrix 
of zeroes) - this assumes that there is an underlying process generating the data points 
which could in principle generate points ad infinitum (see Kott 1991), and 


V isaM X Gmatrix of random variables such that E(V) = Oyxgand E( Vis Viz.) = os(i)- 


It is well known that if U = 0yxG, E(vis¥jr) = Ofori 4 J, and o.(;) = Oo. for all i, then 
Bois = (X'X)“'X'Y (4) 


is the best linear unbiased estimator for 6 (see, for example, Johnston 1972, p. 240). This means 
that the gth column of Bors: call it B.,, is the best linear unbiased estimator of 6. g» where 


5 le Saee, CE oes Tied ia Be (5) 


and y.,, U.,, and v., are the gth columns of Y, U, and JV, respectively. Equation (5) can be 
viewed as the gth equation in the system of equations represented by equation (3). 

Let us call the matrix U in equation (3) the putative missing regressor matrix. Usually, in 
conventional (i.e., model-based) regression analysis that part of the dependent variable (or 
variables) not capturable by a linear combination of the independent variables is (are) assumed 
to be purely random. Here, however, we follow Kott (1991) and allow for the possiblility 
of non-random missing regressors. Note that even when U # Oy yc, Boys remains nearly 
(i.e., asymptotically) unbiased. 


2.2 The (Possibly) Constrained System: 


Efficient estimation is a more complicated matter when there are constraints on some 
elements of 8; for example, when f;, is known to be zero or when 6, ris known to equal ,;. 

In this paper, we are interested in a (possibly) constrained systems of equations that can 
be modelled directly with the following equation: 


y=XBrtrudty, (6) 
where y = (y.j, ¥-35 ---, ¥-G)’, u and v are defined in an analogous manner, X is an 


MG x K matrix, Bisa K x 1 vector, and K < GK. By definition, limy_.X’u/M = Ox. 


When the original 8 in equation (3) is unconstrained, K = GK, and 


X 
D> 
X= 
x 


When the original £ is constrained, however, K < GK. For example, when an element of 6 
is known to be zero, it can be removed from the @ vector in equation (6) along with the column 
of the X matrix that corresponds to it. When two elements in the same row of 6 are known 
to be equal, the second can be removed from 8, and_X can be adjusted accordingly (it will no 
longer be block diagonal). 
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When u = Oyo and Var(v) = ¥ @® Jy (where ¥ = {o,y}), thenbors = (X’X)7'X’y 
is an unbiased estimator for B, but bots = (X’[E7! @ Iw) X) 1X’ [L7! & Iu] y, where 
Iy is the M x M identity matrix, is the best linear unbiased estimator. In practice, the 
elements of ) have to be estimated from the sample, say by 6,7 = r.gr./M, where r., = 
y. 2 = Ux ors: 

It is well known that bo;s and bg;s are equal when the parameter matrix in (3) is un- 
constrained (again, see Johnston 1972, p. 240). Turning to the constrained case, ifu # Oyo, 
then both boys and bgys are nearly unbiased estimators of 8 when limy_.X’U/M = OgygG 
holds as we originally assumed. Unfortunately, bg; 5 may not be nearly unbiased under the 
weaker assumption that limy_.X’u/M = Ox, which is more in line with the extended model 
in Kott (1991) when (6) is viewed as a single equation. 

To see why this is, let X., denote the M x K matrix formed from the {(g — 1)M + 1}th 
through the {gM}th row of X and Y~! = {0}, then 


E(bgts — 8B) « X’L YE © Iylu/M = 


> xi ( Lu ou) | = Lu u a X.4 u[o 


which approaches zero as M grows large under the stronger assumption but not necessarly the 
weaker one. 


3. ESTIMATION WITH SURVEY DATA 


Suppose now that we observe variables values for only a random sample of the population. 
Let P = diag{p;}, where p; is the probability of selection for data point 7. Let S = diag{s;}, 
where s; = 1 if data point iis in the sample and 0 otherwise. Finally, let W = (m/M)P7! S = 
diag{w;} be the matrix of sampling weights, where m is the sample size. When all the 
D; = m/M, note that W = S. 

It is not difficult to show that for many sample designs and populations (see Kott 1990b 
and 1991), the sample weighted OLS estimator: 


Bw-ots = (X’ LIg @ WX) 7X" [Ig © Wily (7) 


is a design consistent estimator for bo, s5, which means that plim,,_.. (Birdos — bors) = Or. 
Under similar conditions, sample weighted GLS estimator: 


(Xe @ WILY ch Ile OX Tie IE LS aay 


By-cLs 


(is OS Wik Oly To ely, (8) 


where 
M 


by = riWes] wy wy jand. r.=)\y —uXBy.o1s: 


i=1 


is a design consistent estimator for boys. Like bo,s and begs (and for the same reasons), 
By-ots and By.cgs are equal for an unconstrained system of equations. 
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If By.ors and By.grs are design consistent, both are also nearly unbiased estimators of 6 
when limy—.X’U/M = Ogyg, because boys and bgys are; however, By.gzs - like bets - 
may not be nearly unbiased under the weaker assumption that limy_.X’u/M = Ox. 
(Unbiasedness here is always defined with respect to the model in equation (6)). 


4. MEAN SQUARED ERROR ESTIMATION 


Suppose the sample design is such that there are H strata, n, distinctly sampled PSU’s 
in stratum h, and m,,; sampled data points in PSU hj. Both By.orssand By-cis-have-the 
form 8 = Cy. Without loss of generality, they can be rewritten as 8 = C* y*, where y* = 
Sin ees nbd contains only elements corresponding to sampled data points, and y;,; is 
the vector of G xX my, y-values associated with data points in PSU hj. Define r* and r,; in 
an analogous manner to y* and y,;. 

Let D,; be a diagonal matrix of 0’s and 1’s such that D,jy* = (0’, ..., nj, ---, 0’), and 
let g,; = C*D,;r*. Extending the design-based linearization variance estimator in a straight 
forward manner, the estimator for the mean squared error of 8 = C*y* has the form: 


H Np Nh 1 nh nh 
mse = ‘25 ; ‘ y 
: joe | ye Ehj Shj al ys su) ( ‘D si) | ( ) 


=] j=l j=l jaa 


Under mild restrictions on the sampling design, mse is nearly unbiased when U (from 
(3)) = OvwxcG and V obeys the following property: 


= 0 whens and ¢are from different PSU’s 
| E( Vso Vif ) | 
< Q otherwise. 


See Kott (1991) for the proof in the G = 1 case; the extension to the G > 1 case is trivial. 
The estimator mse remains reasonable when U # OyxG (See Kott 1990a). 


5. TEST STATISTICS 


Let Bors and @;.crs be the unweighted counterparts of By.o1s and By.cis derived by 
replacing the W in (7) and (8) by S. One is often interested in determining whether using the 
sampling weights really matters. This comes down to testing whether 87.9.5 and By-ors are 
significantly different; that is, whether they are estimating the same thing. 

When weights are determined to matter, another question of some interest is whether 
Bw-ors and By.crs are significantly different; that is, does limyy_.X’U/M = 0gxc hold so 
that these two estimators are estimating the same thing? 


A general statistic for testing whether: 


Bil) = he Ms, {C(1)*Dnrjy* 3 and B 2) = DS Ny {C(2)*Dnj¥”* 3 
h J 


h J 


are equal is 


(eee [Bay a Bay VAT Bay re: Ba], (10) 
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where 


H Np, Np 1 Nh Nh 
sf [Baw -2(B4)(E4)] 


id 


Ang = Cary*Dajtnjay — Ccay*Daj Taj» 204 Taj cpy) = Yay — Xnjp BoA: 

Under the null hypothesis, the test statistic, 7”, is asymptotically a x* random variate with 
K degrees of freedom. Given our concern for robustness, it seems prudent to question the null 
hypothesis when prob(x7x) > 77) is at considerably less than the standard 0.1 or 0.05 level, 
but not when 7” is less than its expected value, K. 


6. AN EXAMPLE 


Consider the following example synthesized from data from the National Agricultural 
Statistics Service’s June 1989 Agricultural Survey. The data set, previously analyzed in Kott 
(1990a), is briefly described below. 

A sample of 17 primary sampling units was selected from among 4 strata. These PSU’s were 
then subsampled yielding a total sample of 252 farms. Although the sample was random, not 
all farms had the same probability of selection. 

Suppose we are interested in estimating the parameters, 6, and 8, of the following 
equation: 


Vile Sip Gist oop bo ee ieee! is (11) 


where / denotes a farm, 

y,; is farmi’s planted soybeans to cropland ratio when i’s cropland is positive, zero otherwise; 
x;; is 1 if farm 7 has positive cropland, zero otherwise; and 

X>; is farm i’s cropland divided by 10,000. 


(Note: dropping all sampled farms with zero cropland from the regression equation will 
have no effect on the parameter estimation, but it can affect mean squared error 
estimation.) 


Letting Bay in equation (10) be the pure OLS estimator for the vector (8,, 62)’, and Ba) 
be the sample weighted estimator, one computes a T of 4.58. Under the null hypothesis that 
OLS and sample weighted least squares are estimating the same thing (for which u,; = 0 is 
sufficient but not necessary), T” is asymptotically X?2) We cannot reject this null hypothesis 
at the 0.1 level. Nevertheless, since T? is considerably greater than 2, it seems that the existence 
of a putative missing regressor is more than likely. Thus, the sample weighted regression 
estimator should be employed rather than the OLS estimator. 

Table | displays both the pure OLS and the sample weighted coefficient estimates. Although 
the sample weighted estimator for , is not significantly different from zero at the 0.1 level, 
we retain it in the model because it exceeds its estimated root mean squared error. This parallels 
the reasoning for preferring sample weighted regression over OLS. 

Notice the loss of efficiency that results from using the sample weighted estimator in place 
of pure OLS. The estimated root mean squared error for the 8, estimator more than doubles 
(note: both root mean squared errors were estimated using equation (9)). 
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Table 1 
Alternative Estimates for Equation (11) 


OLS Vy = 0.268x,, — 0.92x2; + uy; + V4; 
(.044) (3.95) 

sample weighted Yyy = 0.191x4; + 12.15x; + uy + V4; 
(.075) (9.95) 

sample weighted GLS Viz = 0.197x1; + 10.26x2; + uy; + V4; 
(0.71) (6.97) 


Numbers in parentheses are root mean squared errors. 


We can increase the efficiency of the sample weighted estimator by adding a second farm 
equation and estimating it and (11) as a system. Let 


Y2j = X1jB3 + Ug + Vo), (12) 


where jy; is farm i’s planted corn to cropland ratio when i’s cropland is positive, zero 
otherwise. 

The sample weighted estimators in Table 1 and their estimated root mean squared errors 
are unchanged under system-wide sample weighted OLS. The system approach, however, allows 
us to calculate sample weighted GLS estimator for 8; and 8, which are also displayed in the 
table. Observe that the estimated root mean squared error for 6 is reduced by approximately 
30% without a loss of robustness, assuming that sample weighted OLS and GLS are estimating 
the same thing. 

The 7” value for a test comparing the sample weighted OLS and GLS estimators for the 
vector (8,, 82)’ is 0.97. This number is considerably less than 2. Thus, the two estimators do 
appear to be estimating the same thing. That is to say, there is no additional regressor in one 
equation related to the putative missing regressor in the other (which is not surprising since 
when an x>; term was added to the right hand side of equation (12), its estimated coefficient 
was less than its estimated root mean squared error). 


7. SIMULTANEOUS EQUATIONS 


In a simultaneous equation framework, some of the columns of the dependent variable 
matrix, Y, (see (3)) are actually contained on the right hand side of the gth equation (see (5)). 
Formally, we can write 


Nes 1 nO ated Diet bho V A iOfe Y=. Z0 bottle tov, 


where Y(.) = i , 
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Most of the columns of Y,,) are 0-vectors. The rest (no more than G-1 columns) are columns 
of Y from equation (3). 

Define Yo as tA WY (e)- Now replace X in (5) by Z = ( fe ) X) and 
proceed as before. Equation (7) produces ban oLs, akin to two stage least squares, while (8) 
produces Gntora: akin to three stage least squares. Mean squared error estimation follows 
along the same line of reasoning that produced equation (9). 


8. DISCUSSION 


The purpose of this paper was to show how procedures developed in the design-based survey 
sampling literature - in particular, sample weighted regression and the linearization mean 
squared error estimator - could be adopted to the estimation of a system of linear equations. 

One somewhat unexpected discovery was when estimating the parameters of a constrained 
linear system, the sample weighted analogues of OLS and GLS might be estimating different 
things. On further reflection this is not so suprising. If there are missing regressors in our 
working model, perhaps we don’t always know enough about the true model to put constraints 
on the parameters in the first place. 

It is important to realize that mse in equation (9) can be used to estimate the mean squared 
error of parameter estimators even when there are no missing regressors. The advantages of 
mse to conventional practice is that it allows for the possibility of heteroscedasticity and complex 
correlations across data points (but within PSU’s). 

If there are no missing regressors, however, the following estimator has all the advantages 
of mse and is generally more efficient: 


mse’ =; ys | {8aj8nj — 88"/N}, (13) 


h=1 nial 
where n = Yn, and g = Y )'g;; (note: if Xq = (1, ...., 1)’ for some K-vector q, then 
= 0). 


When there are missing regressors the diagonal elements of mse’ may tend to be biased 
upward. The reasoning here follows that in Wolter (1985) for collapsed strata variance 
estimators in design-based sampling theory. 
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The Evaluation of Errors in National Accounts Data: 
Provisional and Revised Estimates 


LUIGI BIGGERI and UGO TRIVELLATO! 


ABSTRACT 


This article is a critical review of recent developments in the evaluation of the reliability of provisional 
national accounts estimates. First, we will sketch a theoretical outline of the process used to produce 
successive estimates of an aggregate, and will reflect upon its implications regarding the design of the 
analyses of errors in provisional data. Second, particular attention will be focused upon the choice of 
elementary measurements of errors and suitable integrative accuracy indices, and the impact of revisions 
on constant price aggregates and implicit deflationary factors. Finally, the results of some empirical 
analysis on discordances between provisional and revised estimates will be summarily discussed on the 
basis of a comparison of national accounts data in Canada, Italy, and the United States. 


KEY WORDS: Revision of data; National accounts; Accuracy indices; Constant price aggregates. 


1. INTRODUCTION 


The evaluation of the reliability of national accounts estimates is important because of the 
effects of these estimates on economic policy analysis and decisions. At the same time, these 
estimates are difficult to evaluate, because they are affected by a very large number of complex 
sources of error. 

Thus, it is difficult to arrive at a single reliability criterion that will be both convincing at 
the conceptual level and practicable. On the other hand, it seems reasonable and feasible to 
establish a certain number of partial criteria that could be used to measure the principal features 
of reliability. For a review of these various criteria, see Novak (1975) and Trivellato (1987). 

The purpose of this paper is to describe and discuss critically certain recent developments 
in the analysis of errors in provisional national accounts estimates. 

National accounts aggregates as well as many other economic aggregates and indicators are 
initially published in the form of provisional estimates that are subsequently subjected to various 
revisions. 

The process of revision is determined above all by the need to ensure that information can 
be made quickly available and, at the same time, by the time required to collect and process 
all the data currently used to estimate the aggregates (‘‘timeliness vs. accuracy,’’ according 
to the effective polarization of Wilton and Smith 1974). Typically, this gives rise to a revision 
process characterized by a preliminary estimate and by a subsequent series of routine revisions 
that follow closely upon one another. From time to time, it may happen that, in order to improve 
the accuracy or relevance of the data, modifications are introduced into the classification and 
accounting plans, and/or in the basic statistical surveys and estimation methods. In turn, this 
leads to new evaluations, in which case, we speak of occasional or extraordinary revisions. 

The characteristic feature of the errors present in provisional estimates resides in the fact 
that they are present in these estimates, and, by definition eliminated in the revised estimates. 
Thus, they can be measured by comparing the two estimates. Strictly speaking, it is true that 
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the difference between a provisional estimate and the corresponding revised estimate is merely 
a measurement of the error difference between the two successive estimates. Nevertheless, when 
the process of revision is based on information that is progressively more complete, and on 
evaluation methods that are increasingly more refined, as is precisely the case in the developed 
countries, it is reasonable to assume that the final estimate is the closest to the true unknown 
value, and for purposes of comparison, to treat it as the reference value. 

An understanding of the characteristics and behaviour of discordances between the provi- 
sional and revised estimates can be truly useful. In fact, significant and frequent differences 
are a discouraging sign respecting the quality of the basic data and/or the estimation methods 
used in the preliminary evaluations, and suggest at least that the results should be used carefully. 
(Naturally, this does not mean that, if the differences are few, this will necessarily represent 
a guarantee of the quality of the data. The fact that a preliminary estimate is not revised may 
simply mean that we do not have enough elements to improve it, regardless of its reliability). 
Moreover, if the differences also seem to be systematic, this may, at the same time, represent 
a helpful warning both to the user, who will eventually introduce correction factors; and to 
the producer, who may derive from this, suggestions to improve his provisional estimates. 

A review of many statistical analyses of the discordance between provisional and subse- 
quently revised estimates can be found in Trivellato (1986a). Other recent contributions have 
been made by Zarnowitz (1982), Mankiw, Runkle and Shapiro (1984), Mankiw and Shapiro 
(1986), McNees (1986), Mork (1987) and Lefrancois (1988). 

In this area, there is a considerable amount of literature. Nevertheless, some questions (both 
methodological and substantive) concerning the way in which these analyses should be carried 
out have been at least partly neglected or resolved in a way that is not always satisfactory. In 
particular, this is true for: 


(a) The identification of a design for the analysis of the accuracy of provisional estimates that 
is consistent with the characteristics of the process of revision; 


(b) The definition of suitable elementary measurements of the discordances between provi- 
sional and revised estimates; the choice of suitable integrative indices that could provide 
adequate information on the overall accuracy of a series of provisional estimates; and, if 
the process of revision is carried out in several stages, of indices that could be decomposed 
in order to be able to verify the convergence of the provisional estimates with the final value. 


(c) The examination of the effects of errors in provisional estimates upon the derived series, 
especially upon constant price aggregates and implicit deflationary factors. 


The three following sections will deal with these problems in order. Finally, section 5 will 
provide a very brief summary of the results of some empirical analyses of the discordances 
between provisional and revised estimates, on the basis of comparisons between national 
accounts data from Canada, the United States, and Italy. 


2. A THEORETICAL OUTLINE OF THE REVISION PROCESS 


A sufficiently general illustration of the process of revision of economic aggregates is shown 
in Table 1, which provides an immediate visualization of the relationships that exist between 
the reference and publication periods of the various estimates. 

Three types of estimates and revisions will clearly emerge from an examination of this 
illustration: 
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(a) those lying on the main diagonal and the lower diagonals describe a current process of 
revision that takes place in m steps; from the preliminary estimate , p, to the final estimate 
r, (final in relation to the current process of revision); 


(b) those of the ,7; type, which lie horizontally, incorporate the benchmark controls where the 
series is reconstructed with reference to a period that goes from one benchmark to the next; 


(c) those of the ,7, type are the result of an occasional revision and also lie horizontally; they 
concern the retrospective reconstruction of the series for a generally rather long period of 
time. 


The figure also shows how the presence of benchmark and occasional revisions that are 
superimposed upon the current revision process results in mixed revisions. Thus, the revision 
that is carried out is not homogeneous with either the previous or the subsequent revisions 
carried out in the current estimation sequence. 

If we have available a chronological series of provisional estimates of an aggregate for times 
1 ton and that of its corresponding revised estimates, the problem of evaluating the accuracy 
of the first in relation to the second can be formally reduced to that of evaluating the validity 
of these forecasts, a problem that has been amply covered in the literature. Nevertheless, 
the mechanism of revising national accounts aggregates, an illustration of which is shown in 
Table 1, has certain characteristic features that we must keep in mind. First, it is important 
to remember that analyses of the process of revision explicitly take into account the existence 
of the three types of revisions listed above. Moreover, in order to understand correctly the 
characteristics of the revisions under consideration, the analyses must also exclude all com- 
parisons that involve mixed revisions. 

A second consideration of particular importance concerns the methods and criteria used 
to analyze the accuracy of provisional estimates. Most studies in the statistical and economic 
literature on the revision of national accounts aim to establish the accuracy of provisional 
estimates on the basis of statistical measurements that are generally descriptive. Even though 
very different, and interesting, approaches have been recently proposed (for example, by 
Mankiw and Shapiro, 1986; and by Lefrancois, 1988), they cannot be entirely satisfactory for 
an analysis of the properties of provisional data. In fact, they represent convenient integra- 
tions, but cannot offer a real alternative to the analysis of the properties of provisional data, 
for which it is important to adopt criteria and accuracy measurements that are essentially 
descriptive. 

Finally, when carrying out an analysis of the accuracy of provisional estimates, it seems 
preferable to treat the provisional and final estimates in a non-symmetrical fashion, and to 
consider the revised series 7; as the reference series. This choice is motivated precisely by the 
fact that we propose to evaluate the accuracy of the provisional estimates on the basis of the 
final estimates (or, in any case, those that are similar to the latter in the comparison under 
consideration). 


3. ELEMENTARY MEASUREMENTS AND INTEGRATIVE ERROR 
INDICATORS IN PROVISIONAL ESTIMATES 


3.1 The Choice Between Errors and Relative Errors 


The error in a provisional estimate (p,) of the level of an economic aggregate can be 
obtained as follows: 


Ve Pry (1) 
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However, this could be inadequate to compare the accuracy of several aggregates, since the 
results depend upon the measuring unit and the order of magnitude of the aggregate under 
consideration. In this case, it may be preferable to use the relative error, which is defined as 
follows: 


Q = (Dr — Wy) /e = D/P, — 1. (2) 


The choice between analyzing the errors or the relative errors can be a crucial one in many 
ways, and deserves a more in-depth discussion. As we will see in section 3.2, in fact, the pertinent 
integrative accuracy indices for provisional estimates are simple averages of v, and e, respec- 
tively (or of their suitable transformations). Thus, the use of simple means is reasonable if the 
series have been derived from a purely random process, particularly if they do not have a trend 
component. In the opposite case, we lose information, and the analysis may lead to obscure 
or even misleading conclusions. For this reason, it is important to carry out preliminary verifica- 
tions, which can be done using various tests (see for example Malinvaud 1969, p. 473-481; 
Kendall 1973, p. 22-28; Box and Jenkins 1970, p. 34-36 and 287-298). 

In relation to the specific problem of choosing between errors and relative errors, a par- 
ticularly useful criterion is offered by the analysis of the parameters of a suitable model of 
the provisional and final data. In its simplest form, this can be specified as follows: 


Dp = a+ Bry + Ss (3) 


where €, is the random error. From model (3) we can obtain: 


Ve = at AB =) ees (4) 
1 Ey 

ej = (6 —1) +a—+—. (5) 
rt ry 


Formulae (4) and (5) show that, generally, both the error and the relative error of the pro- 
visional estimate depend upon the level of the corresponding final estimate. From these we 
can then derive the conditions that must be satisfied in order to justify the use of integrative 
indices based on the errors or relative errors respectively: 


(a) v,does not depend upon 7, if 8 equals 1, and the €,s are homoscedastic, and not temporally 
correlated; 


(b) e, does not depend upon r, if a equals zero, and the variance of the €,s is proportional to 
the square of the level of the series (the €,s are not temporally correlated). 


It will be immediately clear that estimating (4) with ordinary least squares is equivalent to 
estimating (5) with generalized least squares ( with E(€€’) = 020, where Q is a diagonal 
matrix with w,, = 77). It is also immediately evident that (3) and (4) differ only by 1 in the 
angular coefficient. Thus, in order to choose between analyzing the errors or the relative errors, 
it becomes crucial to verify the homoscedasticity of €, in (3). In order to do this, Trivellato, 
Di Fonzo, and Rettore (1986) developed a simple non-parametric test based on the order of 
the estimated residuals, which does not depend upon specifying a particular stochastic structure 
for equation (3); this conforms with the little we know a priori about the relationship that exists 
between provisional and final estimates. 
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This test can also be used to examine the hypothesis of the stability of the parameters in 
equation (3). In fact, it is evident that conditions (a) and (b) above are not exhaustive, and 
that a possible reason for their lack of validity is the presence of instabilities in the revision 
process. The importance of carrying out this type of verification is based, among other things, 
on the fact that we study the superposition of occasional revisions onto the current revision 
process. If the results of the test favour the hypothesis of stability, it is appropriate to analyze 
the current revision process as a whole. In the opposite case, the analysis must be carried out 
separately for the two periods preceding and following the occasional revision. 

Finally, it is important to point out that specification (3) is intentionally stylized and can 
be essentially used to explore the process of revising an annual series, when the residual €,;s 
are not self-correlated. Nevertheless, it is easy to generalize it in various ways: (i) by introducing 
a vector of non-random explanatory variables in order to take into account any eventual factors 
that may affect the process of revision (benchmark revisions, seasonal factors, efc.); (ii) by 
assuming that the residuals are self- correlated; (iii) by modelling jointly the process of revision 
of several series using a seemingly unrelated regression equations system. For these 
developments, see Trivellato and Rettore (1986), and Bordignon and Trivellato (1989). 


3.2 Integrative Indices of the Accuracy of Provisional Estimates and ‘‘Low Coherence’’ 


In order to characterize suitable integrative indices of the overall accuracy of a provisional 
estimates vector, it is important to refer to the property of ‘‘low coherence,’’ as defined by 
Trivellato (1986b). In substance, in reference to two series of provisional estimates correspon- 
ding to the same series of final estimates, this requires that if the first series shows errors that 
are smaller than or equal to those of the second in absolute value, and if significant inequality 
is present in at least one case, the accuracy index of the first series must be smaller than the 
index of the second series, and thus indicates that the first series of provisional estimates is 
closer to the final data than the second. 

As we will see, the central reference to integrative low coherence indices does not lead to 
any significant innovations in relation to the normally used measurements (we will essentially 
use the mean absolute error and the mean quadratic error). This concept, together with the 
choice of analyzing either the errors or the relative errors, and the eventual identification of 
homogeneous sub-periods in the revision process can nevertheless lead to an adequate 
implementation of empirical analyses, while avoiding the lack of accuracy that can often be 
found in the literature. 

It is also important to consider the decompositions of the low-coherence integrative indices. 
From a slightly different point of view, the presence and weight of specific components can 
eventually be revealed by looking at the estimated parameters in equation (3) (Mincer and 
Zarnowitz 1969; Hatanaka 1974; Hempenius 1980; Trivellato 1986a). 

The indices that we should emphasize are the following: 


(1) Errors 


If the criteria used to choose between the errors and the relative errors (see section 3.1) lead 
to the implementation of an analysis of errors, two low-coherence integrative indices would 
be the mean absolute error and the square root of the mean quadratic error: 


v= Ly /n, (6) 
dy = N( Yviin); (7) 


where the sum is extended to n terms in the series. 
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The following decomposition, similar to Theil’s asymmetric decomposition (1966), but 
developed by treating 7, as a reference series, makes it possible to show the deficiencies in the 
performance of provisional estimates: 


d} = (p — F)* + (s, — ps)? + (1 — 6”) 8, (8) 


where s, and s, are the standard deviations of r,; and p; respectively, and 6 is the coefficient 
of correlation between p, and r,. From formula (8) we can derive the following relative decom- 
position: 


1=U" + UF + UP. (9) 


The three terms in equation (9) represent fractions of the mean quadratic error and can be 
interpreted as follows: a bias component between the means of the two series, U™; a regres- 
sion component attributable to a deviation of 1 from the regression coefficient of p,; over 
r,, U8; and a random error component attributable to the variance of the regression errors, 
Oped 

When assessing the quality of estimates, it is also useful to take into account the mean and mean 
quadratic deviation of the errors; that is, > = Yv,/n = p — Fands, = VY (vy, — 9)*/n) 
respectively. Clearly, 9 is not an index of the accuracy of provisional estimates. It provides 
information only about the direction and dimension of the mean level error. Nevertheless, it 
is a Statistical measurement of remarkable significance for at least two reasons: (i) if ¥) = 0, 
then there is no bias component (U™” = 0); (ii) if 5 ¥ 0, it will be instructive to examine 
¥and ¥’ jointly, since | } | = ¥’ if and only if the errors are always (or almost always) in the 
same direction. Thus, the comparison between the two indices can show the presence of a 
possible systematic component in the level errors of the provisional estimates. 


(II) Relative Errors 


When the criterion used in section 3.1 leads to the analysis of the relative errors, two suitable 
integrative indices are the absolute mean relative error, and the square root of the mean 
quadratic relative error respectively: 


e = Yle| /n, (10) 
d, = V(¥ez/n). (11) 


The two indices are defined if r, # 0 for each t, a condition that remains non-restrictive for 
most economic aggregates. 


A reasonable decomposition of d, is as follows: 
2 ar: 2 rte 
de = [(p/r) — 1])* + jal Belted Tel PAG dr» (12) 


from which can be derived the relative decomposition 
1 = U? +.US, (13) 


U? being the fraction of the mean quadratic relative error due bias, and US the fraction due 
to the random component. 
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Two other specific components of the lack of accuracy should also be mentioned; these are, 
the mean and mean quadratic deviation of relative errors; that is, @ = Ye,/n = (p/r) — 1 
ands, = MW) ¥ (e, — &)2/n) respectively. Mutatis mutandis, what we have said for 9 is also 
valid for é, both in terms of its meaning and for the comparison with é’. 


3.3 Decomposition of the Revision Process from the Preliminary to the Final Estimate 


The accuracy indices discussed until now involve a comparison between two vectors p and 
r. However, as we have already observed, the process of revision of economic aggregates cannot 
usually be completed in a single stage. Thus, how can we summarily estimate the convergence 
of the succession of provisional estimates with the final estimate? 

In this section, we will discuss the decomposition of a process of revision that takes place 
in two stages (a procedure for the most common situation of m — 1 steps, can be found in 
Wilton and Smith 1974). This decomposition will be evaluated with reference to (i) errors; and 
(ii) relative errors respectively. 

When we analyze the errors, the decomposition of the mean quadratic error of ; p in rela- 
tion to rin the two phases from ,p to »p and from jp to ris easy to obtain. Let ;v = ;p — 2p, 
>v = 9p — Fr, rv = 1p — rand let d2,, d*,, d?show the mean quadratic error associated 
with vectors ;v, »v and ;v respectively. The result is: 


2, 
dir = ad, + diy + é LY i% 2M» (14) 


from which we derive the relative decomposition: 
1 = D, + Dy + Dyn, (15) 


where the first two components represent the fraction of d*7 that can be attributed to the 
mean quadratic error of the first and second revision respectively, while D, 7; represents the 
interaction between ,v and 4v. 

On the other hand, if we consider the relative errors, an integrative decomposition of the 
process of revision in the two stages may be problematic. Useful indications can be derived 
from the equation (;p; — ¢)/T; = (:Pt — 2Pt)/1t + (2Pt — 11)/13 thus, by dividing the 
two members by (,p; — 7;)/r; and taking into account the mean values, we obtain: 


1 1 
l= me. (EDerD: GPE Tp et 7 GPraty7 (epi ras (16) 


If we assume a stable order of estimates of the ;p,type < 2p, < r, for each?, the two terms 
of equation (16) can be interpreted as mean fractions of the discordance between ;p andr 
eliminated by the first and second revision respectively. Nevertheless, this interpretation can 
be questioned in terms of order violations, especially when this occurs together with small 
1P; — 1; differences. In general, it seems more reasonable to discard the hypothesis that the 
order between estimates must be respected for the whole period under consideration. In this 
situation, a qualitative evaluation of the degree of stability of the order relationships between 
the estimates, and the importance of the two stages in the revision process can be obtained 
from an examination of values of é and @’, together with comparisons between ,p and »p, 
between 5p and r, and between ,p and r. Clearly, the relationship | @ | = @’ is valid if and only 
if the order between the two series of estimates involved in the comparison is always respected. 
However, we can say that the order is ‘‘most often’’ respected when | é | = @’ meaning that: 
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(2) =o) 2] )/2’ <2f, (17) 


where f is a fraction of violations of a predetermined order, that is considered to be acceptable. 


Equation (17) can be justified as follows: let e,(¢ = 1, 2, ..., m); the relative errors, 7), 
are strictly positive; 12s are strictly negative; and n3s equal zero (nm; + nm. + n; = n). It is 
easy to verify the following relation: 


On ery eee itl Mele 
od 27a tay 1 & if é> 0, 


where quantities 5 ?!, e,and — y % j=1 & correspond to the absolute value of the sum of the sign 
violations of é. 


The quantity 
a) pee eh OMe ej — yj214e) lif pe a0 


say pp bee rere Ler e)) ie <0) 


(2, -— e+) /e.-= 


is thus an index of the importance of the sign violations of é. This Boh is bounded as follows: 
0 < (é’ — | @|)/é’ < 1; the lower limit is reached when | @ | = @’; that is, whenn, = 0, 
or when n, = O(if nm, = ny = 0, there are no order violation ona and the upper limit 
is reached when @ = 0; that is, when Y 71, e, = D721 6. 

The relationship | é@ | = @’ is defined by comparing the value assumed by the index with 
a critical value calculated in accordance with the hypothesis of the equality, in absolute value, 
of all the non-zero relative errors. This produces the following result: 


2n,/(n, = nN») if @€<0 
(é’ — |é|)/e’ = 
2n,/(n; + Mm) if @> 0. 


Under this particular hypothesis, the index is thus equal to twice the fraction of strict viola- 
tions of the sign of é. 

Thus, it is possible: (i) to rank the estimates , p and 5p on the basis of a mean criterion; that 
is, the values of @ associated with the various comparisons; (ii) to examine the degree of stability 
of the order on the basis of equation (17); (iii) to use decomposition (16), or to estimate 
qualitatively in any other way, the magnitude of the two stages, by observing the values of 
é associated with the various comparisons, when there is a high degree of stability. 


4. THE IMPACT OF ERRORS IN PROVISIONAL ESTIMATES 
UPON CONSTANT PRICE AGGREGATES 


Evidently, the revision process has certain effects upon the derived series, which often contain 
the information that holds the greatest interest for analysts and policy-makers. This is partic- 
ularly true in the case of estimates of the rate of variation, seasonally-adjusted series, constant 
price aggregates, and implicit deflationary factors. The impact of revisions on measurements 
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of variation has been analyzed descriptively by Trivellato, Di Fonzo and Rettore (1986); and 
by Rao, Srinath and Quenneville (1989), who attempted to determine the best estimator of 
variation. The effects of revisions upon the seasonal adjustment procedures have been described 
in papers by Pierce (1980), Wallis (1982), and Maravall and Pierce (1983). The effects of the 
revision process upon constant price aggregates and implicit deflationary factors will be 
examined in this section. 

The criteria and indices discussed in section 3.2 could evidently be used to analyze the 
accuracy of provisional constant price estimates. It would nevertheless be interesting to illustrate 
the formal characteristics of errors in these estimates, and to show the relationships that exist 
between estimation errors in current price and constant price aggregates, as well as the implicit 
deflationary factors. The emphasis will be solely upon aggregates of national economic accounts 
made up of the flow of goods, and the aggregates obtained using accounting balances for these 
goods. 

Let us consider the simple case of a series of provisional estimates and the corresponding 
series of final estimates. By observing any aggregate at time ¢, consisting of k elementary goods 
and services, the final estimates can be obtained as follows (when we have a sum of k elementary 
goods and services): A, = ¥p;q;is the current price aggregate; C; = ) p,q, is the constant 
price aggregate (0 base); P, is the Paasche price index (that is, PR. = ¥p,q;/¥ Po 41); Q; is the 
Laspeyres quantity index (that is, O, = Ypoq:;/¥ Po Qo); D; = Ar /C; is the implicit defla- 
tionary factor with a Paasche structure. The corresponding provisional estimates are identified 
with the subscript p (thus, for example, ,A; and A, are respectively the provisional and final 
estimates of the current price aggregate). 

In order to explain the characteristics of the errors in provisional estimates of constant price 
aggregates, it would be useful to remember that the evaluation of the constant price aggregate 
at time tf may be obtained using three methods: (i) if we have a series of quantities for all the 
goods and services that make up the aggregate and the corresponding prices for the base year, 
we use the direct relationship: C,; = ¥ p,q;; (ii) if we have a Laspeyres-type quantity index, 
we multiply the value of the aggregate in the base year by this index: C; = (Y¥ Pogo) Q;3 
(iii) if we have a Paasche-type price index, we divide the current price aggregate by this index: 
C, = (LP14:)/ Pr. 

In relation with the various useable evaluation criteria, the error in the provisional level 
estimate (as well as the relative error) of a constant price aggregate can be obtained as follows: 


LPopG LY Po 4 
pr — Ce = L Pop — L Pot = Epode( oe = 
Y Dede Pao 


) = Ao(pQ ='Q;), (18.1) 


= (LY Po Qo) pQ: ist (Y Po Qo) QO; er Ao (pQ aT Q,), (18.2) 


pArl Ps — A,/P, = (DL pPi parla: Spray) (Ps (18.3) 


Equations (18) show that, if we adopt the direct evaluation criterion (18.1), or extrapolation 
by means of an index of quantity (18.2), the estimation error coincides with the estimation error 
in the index of quantity, multiplied by constant A,. On the other hand, this result cannot be 
obtained using the other indirect deflation criterion a price index, because, in general, (18.3) 
cannot be simplified in a way that is useful for our purposes. In other words, in this third case, 
the error in the estimate of the level of the constant price aggregate generally depends both 
upon errors in the provisional estimates of quantities and upon errors in the provisional 
estimates of prices. 
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The implication of this for the interpretation of errors in the estimates of constant price 
aggregates can be quickly made clear. Even though, theoretically, the three evaluation criteria 
are identical, this is far from being the case in practice, due to the availability and quantity 
of data. It is important to remember that evaluation at constant prices is generally obtained 
and a very desaggregated level, and that the aggregates are subsequently obtained by adding. 
These aggregates are thus evaluated with the deflation criterion, errors in the constant price 
estimates are not only errors in the volume estimates , but also contain errors in the price 
estimates. 

If we now consider the relative error in the estimate of a current price aggregate and make 
explicit its relationship with the relative error of the corresponding constant price aggregate, 
we have the following equation: 


pAt = A, por = C; pu: rs D; A OF We C; ps a D; 
= +f +g —__ >x —__., 
A; C; D; C; D; 


This equation is evidently valid for the mean relative error of a series of n estimates, and we 
can obtain the following relative decomposition: 


Pi C(C)/e( 4 jarere( D7 Cl A) tre (C. D)7e(A), (19) 


x whi: Agim At 
@(A) = - > Sse ) 


(@€(C) and @(D) are defined in a similar manner, and 


1 ale -—C¢, pp me: 
XCD). = = —_—_— ——_—— }.. 
B spine n Xe ( C; ) ( D, ) 


If we leave aside the interaction component, we will then have an approximately additive 
decomposition: the mean relative error in the current price aggregate is equal to the sum of 
mean relative errors of the constant price aggregate and the implicit deflationary factor (or 
better still, if we take into account the estimation process: the mean relative error of the implicit 
deflationary factor can be approximately obtained by the difference between the mean relative 
errors of the current price aggregate and constant price aggregate respectively). 

It is important to emphasize that the decomposition is between @(C) and @(D), and not 
between the ‘‘error in the quantities’’ and the ‘‘error in the prices.’’ We have already seen that, 
in general, ,C, — C, and @(C) also reflect any errors in the provisional price estimates. On 
the other hand, by definition, ,D, — D,, and é(D), area function of errors in the provisional 
estimates both of prices and of quantities. Thus, at the end, both components, ‘‘the error in 
the constant price estimate’’ and ‘‘the error in the estimate of the implicit deflationary factor’’ 
incorporate errors in the estimates of price and quantities; and the possibility of interpreting 
pC: — C,and ,D, — D, as “‘error in the quantities’’ and ‘‘error in the prices’’ respectively can 
be relegated only to extreme cases. 


where 


We can identify the following extreme cases: 


(a) »C, — C;, can be interpreted as ‘‘an error in quantities’? only when the evaluation of the 
constant price aggregate has been carried out with the direct criterion or by extrapolation 
using an index of quantity; 


110 Biggeri and Trivellato: Evaluating Errors in National Accounts Data 


(b) ,D, — D, can be interpreted as ‘‘an error in the prices’’ only in the absence of a revision 
in the quantities. In this case, the error in the constant price estimate is evidently zero, and 
the relative current price error can be expressed as a linear combination of estimation errors 
in the prices of k elementary goods and services: 


(pA; — Ar)/Ar = (pD; — Dr)/Di = LG: (Pt — Pt)/ UVP = La(pPr — Pr), 


where 
Qa = G/L P14 - 


(c) If there are no revisions in the prices, we have: 
(pAr — At) /At = Ldi (pGe — Gr)/ UDG = (pCr/Cr) (pDr/Dy) — 1. 


The absence of revision in the prices is not enough to distinguish between ‘‘errors in 
quantity’’ and “‘errors in prices,’’ since the revision of quantities modifies the implicit 
deflationary factor. 


5. SOME EMPIRICAL ANALYSES ON REVISIONS OF NATIONAL ACCOUNTS 
DATA IN CANADA, THE UNITED STATES AND ITALY 


5.1 The Process Used to Revise National Accounts Data in the Three Countries and the 
Analyses Carried Out 


In this last section we will summarize some of the results of empirical analyses on the revisions 
of national accounts in Canada, the United States and Italy. 

The comparison of the characteristics of the process of revision used in the three countries 
contains a certain number of inevitable simplifications in the description and analyses of the 
revisions. For recent in-depth studies on each of the countries under consideration, refer to 
Lefrancois (1988) for Canada, Parker (1986) and Mork (1987) for the United States, and 
Di Fonzo, Rettore and Trivellato (1986) for Italy. 

Charts of the processes used to revise national accounts data on a yearly and quarterly basis 
in the three countries under consideration are shown in Tables 2, 3 and 4. 

The empirical analyses were carried out on the following data: 


(a) Canada: quarterly non-seasonally-adjusted estimates and annual current price estimates, 
for the period between 1953 and 1982. This is the usual database used by Statistics Canada 
to analyze revisions. 


(b) United States: quarterly seasonally-adjusted estimates for the period between 1968 and 
1983. In this case also. this is the base normally used by the Bureau of Economic Analysis 
(BEA) to analyze revisions; however, contrary to the Canadian data, these do not coincide 
with the published data, because the BEA introduces adjustments and corrections to 
eliminate the effect of changing definitions. 


(c) Italy: annual estimate series, for the period between 1961 and 1985 (we did not take into 
account quarterly estimates, because publication was started in 1976, but was discontinued 
in 1982). These are data published by the Jnstituto Centrale di Statistics (ISTAT). 


By taking into account available data, it is possible to carry out comparisons between Canada 
and the United States for the quarterly process, and between Canada and Italy for the annual 
process. 
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Table 2 


Publication Plan for Quarterly and Annual Estimates of National 
Accounts Aggregates in Reference to Time ¢ in Canada 
(Current Revisions Only)? 


Period of Period of Reference 
Publication aaa, 
Year Month and 2) Rea oa 
Quarter I II III IV 
t 1 
Zz 
3 
4 
5 
6 
7 
8 
9 


janh fom 
— =) 


1 
2 4PrI 3PLu 1Pr.Iv iPr 

3 
Ul 2Puv 2B 

Ill 

IV 

t+2 I 
II rey Te Tea ThIv r; 

Ill 

IV 


4 Series shown are those used to carry out the empirical analysis on the revisions of 
quarterly data. 


In order to carry out comparative analyses between the processes used to revise quarterly 
estimates in Canada and the United States, it was necessary first to establish which of the U.S. 
series could be used in parallel with the Canadian series. By taking into account the temporal 
shift between successive estimates, the available data, and the characteristics of the revision 
process, we chose the series in Table 3. In this way, we have, for the two countries, and for 
each reference quarter, three estimates that will be identified by the same letter: P is the first 
quarterly estimates series, RT is the chosen series of quarterly revisions; and A1 is the first 
annual revisions series. To these three estimates, we added the final estimates; these coincide 
with the last published estimates, which we will identify with the letter F. 


Subsequently, it seemed appropriate to carry out the following comparisons: 


(P,RT), which takes into account the effect of the quarterly revision (sub- annual). Never- 
theless, it is important to point out that, in Canada, only the provisional data for the first 
three quarters of any year are subject to strictly quarterly revisions, since the revisions to 
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the fourth quarter data (2); j, in Table 2) is made up of both a quarterly and an annual 
revision. Among other things, this results in a very obvious difference in the behaviour of 
the revisions (see Lefrancois, 1988). We took this characteristic into account simply by 
limiting the analysis to the first three quarters of every year. 


(RT,A1), which provides information on the contribution made by the annual benchmark 
to the quarterly process of data revision; 


Table 3 


Publication Plan of Quarterly and Annual Estimates of National 
Accounts Aggregates in the United States* 


Period of Period of Reference 
Publication TTR 
ear Month and prices (6 RUAN Sistineptiah Nags! 
Quarter I II Ill IV 

t 1 
2 

3 

4 

5 

6 

7 

8 

9 

10 

11 


1 
2, 
3 3PL.IVv 
II 
U1 2 
IV 
t+2 I 
Il 
Ill 5Pr.1 sPru 5 Pri 5Pt.iv 2Pr 
IV 
t+ 3 I 
II 
Ill rey rey Tet Try r; 
IV 
Ae IV ol Lt eat bl Ll bl Liv ot 
t+ 10 IV oli lL il ole it bl Liv blr 


@ The series shown are those used to carry out the empirical analysis on the revisions of 
quarterly data. 


Survey Methodology, June 1991 113 


Table 4 


Publication Plan for Quarterly and Annual Estimates of National 
Accounts Aggregates in Italy 
(Current Revisions Only) 


Period of Period of Reference 
Publication 
Quarterly Data 
Month and nn Yearly 


Year Quarter I I Ill IV Data 


1P1.1 


Omaoarrytnan fh WN — 


1Prit 


— 
So 


— jm 
Ne 


1Pri1 
t+ 1 


2Pr.1 2Pt.0 2Pt.A0 1Pt.i1Vv 1Pr 


t+2 I 3Pr.1 3Pr.11 3Pr.11 2Pt.1V 2Pr 


bestia I Tey Te Tet Th iv ry 


(A1,F), on the basis of which we were able to evaluate the effect of five-year benchmark 
estimates in the United States and compare them to those determined by the extraordinary 
revision process in Canada. In this respect, we should also point out that the comparison 
must be carried out with a great deal of caution, since the Canadian historical revision are 
mainly caused by changes in the definitions, and are not strictly comparable with the five- 
year data adjustment process carried out in the United States. 


As far as the comparison between current revisions of annual data in Italy and Canada is 
concerned, we still have some minor problems, because in the two countries the data available 
describe a process of revision that takes place in two stages. We identified the three series of 
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estimates under consideration in the usual way, as follows, ,;p, »p and r; in this case, the 
interesting comparisons are: (,p,r), to evaluate the overall effect of the current revision 
process; (,p,r), to verify how much of the revision is carried out in the second stage (and 
indirectly also to derive information on the first stage); and the simultaneous comparison 
(,P.2p,r) to evaluate the convergence of the series of estimates. 


The analyses were carried out on the following four aggregates: 
PL = Gross Domestic Product for Italy; Gross National Product for Canada and the United 
States. 
CF = Final domestic consumption of households and private administrations for Italy; 
personal expenses for goods and services for Canada and the United States. 
CP = Collective consumption of public administrations for Italy; Acquisition of goods and 
services by public administration for Canada and the United States. 
IN = Gross Fixed Capital Formation. 
It should be kept in mind that the differences between the reference accounting systems used 
(SNA for Canada and the United States, SEC for Italy) mean that the configuration of the 
aggregates is not completely homogeneous in the three countries. 
The main results of the analysis of errors in estimates of the level of the aggregates are shown 


in Tables 5 to 8. More complete results are nevertheless available and we will refer to them 
as the occasion arises. 


5.2 A Tentative Comparative Summary of the Results of the Analyses 


A comparison of the three plans shown in Tables 2, 3, and 4 leads to the following 
conclusions. 


Table 5 


Indices of the Accuracy of Provisional Estimates of the Level of 
Current Price Aggregates - Canada 


Aggregates é é’ ie d. ue 
Comparison between P and RT (1953-83; T = 89) 
PL — .0018 .0034 .0042 .0046 .1567 
CF — .0009 .0025 .0039 .0040 .0560 
CP — .0041 .0106 .0193 .0197 .0425 
IN — .0047 .0133 .0206 .0211 .0498 


Comparison between RT and A1 (1953-83; T = 69) 


PL — .0046 .0064 .0065 .0080 LY PA 
CF — .0054 .0069 .0086 .0102 .2877 
CP — .0008 .0176 .0262 .0262 0009 
IN — .0075 0205 .0270 .0280 .0721 


Comparison between A1 and F (1953-70; T = 72) 


PL S050 .0556 .0197 .0590 .8882 
CF — .0617 .0617 .0192 .0646 .9110 
CP — .0673 .0727 .0585 .0892 5699 


IN — .0228 0311 .0326 .0398 .0328 
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Table 6 


Indices of the Accuracy of Provisional Estimates of the 
Level of Current Price Aggregates 
Annual Data - Canada 


Aggregates é e’ S35 de ye 
Comparison between ,p and r (1953-83; m = 20) 
PL — .0084 .0093 .0058 .0102 .6720 
CF — .0080 .0086 .0077 .0112 5191 
CP — .0069 .0118 .0136 .0153 .2052 
IN — .0173 .0183 .0114 .0208 .6966 
Comparison between »p and r (1953-83; n = 24) 
PL — .0032 .0041 .0044 .0054 3444 
CF — .0029 .0030 .0048 .0056 .2619 
EP — .0038 .0063 .0087 .0096 .1627 
IN — .0040 .0042 .0075 .0085 .2274 
Comparison between r and F (1953-70; n = 18) 
PL — .0464 .0464 .0188 .0500 .8588 
CF — .0544 .0544 .0200 .0580 .8809 
CP — .0523 .0523 .0348 .0629 .6928 
IN — .0130 .0147 .0140 .0192 .4637 
Table 7 
Indices of the Accuracy of Provisional Estimates of the 
Level of Current Price Aggregates - United States 
Aggregates é e’ Se de Ue 
Comparison between P and RT (1968-83; T = 64) 
PL — .0017 .0031 .0040 .0044 .1459 
CE — .0014 .0041 .0054 .0055 .0607 
CP — .0013 .0050 .0064 .0065 .0433 
IN — .0047 .0104 .0138 .0146 .1052 
Comparison between RT and A1 (1968-83; T = 52) 
PL — .0054 .0064 .0071 .0089 .3706 
CF — .0049 .0073 .0078 .0092 .2854 
CP .0001 .0088 .0109 .0109 .0002 
IN — .0074 .0176 .0237 .0249 .0878 
Comparison between A1 and F (1968-83; T = 52) 
Pr — .0093 .0097 .0070 .0117 .6411 
CF — .0040 .0070 .0088 .0096 .1662 
cP .0039 .0070 .0088 .0096 .1662 
IN — .0408 .0409 .0253 .0480 yaw 
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Table 8 


Indices of the Accuracy of Provisional Estimates of the 
Level of Current Price Aggregates 
Annual Data - Italy 


f 


Aggregates é é Se d. ue 


Comparison between ,p and r (1963-85; n = 13) 


RE ='.0095 .0095 0093 .0133 5091 
CF — .0067 .0068 .0057 .0088 5781 
CP. — .0111 .0122 .0129 .0170 .4249 
IN — .0062 .0078 .0091 .0110 3141 
GR.1 — .0032 0113 .0140 .0144 .0489 
2 —.0253 .0282 0283 .0380 .4443 

5 20107 .0379 .0462 .0475 .0508 

o 20133 .0208 .0271 .0302 21937 

5 — .0056 .0101 .0113 .0126 .1994 

6 — 0044 .0172 .0206 .0210 .0429 

7 — .0143 .0146 .0150 .0207 4750 

8 — .0061 .0132 .0170 .0181 LQ 
Total — .0106 .0106 .0106 .0150 4987 


Comparison between »p and r (1963-85; n = 17) 


ied 8 — .0030 .0037 .0054 .0062 neoe 
CF ='.0015 .0017 .0024 .0029 .2900 
cr = OOLS .0049 .0063 .0064 .0426 
IN = 3UULE .0026 .0052 .0056 1445 
GR.1 .0002 .0026 .0038 .0038 .0018 
Zz — .0054 .0096 .0172 .0180 .0884 

S. = .0085 .0096 .0177 .0177 .0073 

= — .0058 .0060 0105 .0120 .2304 

> .0008 .0016 .0026 .0027 .0954 

6 .0011 .0055 .0085 0085 .0167 

7 — .0039 .0069 .0093 .0101 .1459 

8 — .0008 .0008 .0030 .0031 .0673 


Total — .0026 .0038 .0058 .0063 .1648 


Gr.1: Agriculture, food and beverage products, tobacco. 

Gr.2: Textiles, clothing and shoes, skins and leather. 

Gr.3: Wood and wood furniture. 

Gr.4: Non-metal minerals, chemical products, extraction industries, energy and water. 
Gr.5: Buildings and civil engineering. 

Gr.6: Transportation and communications. 

Gr.7: Other areas. 

Gr.8: Building rentals. 


The system used to produce and revise national accounts estimates in the United States seems 
to be much more structured and strongly oriented in favour of quarterly accounts; in this 
respect, it is important to emphasize that the preliminary estimates are published extremely 
fast. By the time Canada (and Italy) produce the first quarterly estimate, in the United States, 
this estimate has already been revised two or three times using a remarkable amount of addi- 
tional direct information (Parker 1986). On the contrary, less importance is given to the annual 
estimate, which comes out with a delay of nine months. 
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On the other hand, speed seems to be a distinctive characteristic of Italian annual estimates, 
which come out during the first quarter of the next year; while in Canada, the first annual esti- 
mate is only available in the second quarter (the preliminary estimate published in the first quarter 
is a simple aggregation of quarterly data). However, this characteristic is not necessarily a good 
thing, and it is reasonable to think that the fast production time of the initial annual estimates 
can also be explained on the basis of gaps present in the Italian quarterly accounts system. 

Finally, it is important to emphasize that the United States is the only country that carries 
out not only regular annual benchmark revisions for quarterly estimates, but also more long- 
term benchmark revisions based on input-output tables, and the results of wider surveys, for 
example, a census. This ensures greater coherence (and not only from an accounting point of 
view) in the estimates of the various aggregates. In Canada and in Italy there are similar revi- 
sions; however, these are only carried out occasionally and are almost always accompanied 
by modifications in the classification or evaluation criteria. 

On the basis of Tables 5 to 8, we are able to make the following observations: 


(a) In the Canada-U.S. comparison, the pattern of current revisions of quarterly estimates 
is substantially homogeneous, at least as far as the aggregates in question and the limits 
of the revision process described by the three available series of estimates are concerned. 
On the average, sub-annual revisions are rather modest (in both cases the JN aggregate 
is an exception) and their behaviour is rather irregular, even though the trend is to revise 
upwards. The effect of the annual benchmark revision seems to be relatively more marked 
and systematic, especially in terms of the PL. There is also a tendency to correct the previous 
evaluation upwards. In this second stage of the process of revision, aggregate IN also shows 
higher and more irregular size errors. 

(b) In relation to annual estimates, we also found substantial behavioural similarities in the 
Italy-Canada comparison. On the average, the preliminary PL estimate underestimates the 
final current data by about 0.9%; this trend is very systematic and has a sufficiently stable 
order so that ;p < >p < r. This leads to an underestimation of the rates of variation 
when they are calculated (as is generally done) on the basis of ‘‘horizontal’’ comparisons. 
We should add that the weight of the second revision is relatively more marked for Canada. 


On the whole, the empirical evidence shows that the provisional estimates are valid, especially 
if we look at quarterly evaluations in Canada and the United States. On the other hand, the 
systematic character of the underestimations of preliminary annual estimates, both in Canada 
and in Italy, raises certain questions (even though the size of the error is certainly not significant). 

This phenomenon is widespread in many countries (see Glejser and Schavey 1974). There 
are nevertheless some aspects of the Italian case that deserve further consideration. The first 
are the extreme differences in the behaviour of errors at the disaggregated level by area of eco- 
nomic activity, which points to situations that are certainly weak from the point of view of 
information: in some areas, there is a systematic underevaluation of 6-7% on the average 
(Marliani 1986). 

The other aspect is related to the marked re-evaluation effect due to the extraordinary 
revisions, which generally affects all aggregates (see Di Fonzo, Rettore, and Trivellato 1986). 
The trend to revise the previous evaluations upwards when introducing statistical modifications 
in the basic statistical surveys and/or the estimation methods, is certainly not unique to Italy. 
The Canadian data (and indirectly, the U.S. data, through the effects of the five- year bench- 
mark adjustments) show a similar trend. 

These are signals that emphasize the weakness of national accounts systems, and lead us 
to consider with less optimism the small average size of the errors in provisional estimates 
generated on the basis of the currently used revision process. 
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GUIDELINES FOR MANUSCRIPTS 


Before having a manuscript typed for submission, please examine a recent issue (Vol. 10, 
No. 2 and onward) of Survey Methodology as a guide and note particularly the following 


points: 

i Layout 

1.1 Manuscripts should be typed on white bond paper of standard size (8% x 11 inch), 
one side only, entirely double spaced with margins of at least 1% inches on all sides. 

1.2. The manuscripts should be divided into numbered sections with suitable verbal titles. 

1.3. The name and address of each author should be given as a footnote on the first page 
of the manuscript. 

1.4 Acknowledgements should appear at the end of the text. 

1.5 Any appendix should be placed after the acknowledgements but before the list of 
references. 

jae Abstract 
The manuscript should begin with an abstract consisting of one paragraph followed 
by three to six key words. Avoid mathematical expressions in the abstract. 

a Style 

3.1 Avoid footnotes, abbreviations, and acronyms. 

3.2 Mathematical symbols will be italicized unless specified otherwise except for functional 
symbols such as “exp(-)” and “‘log(-)’’, etc. 

3.3 Short formulae should be left in the text but everything in the text should fit in single 
spacing. Long and important equations should be separated from the text and numbered 
consecutively with arabic numerals on the right if they are to be referred to later. 

3.4 Write fractions in the text using a solidus. 

3.5 Distinguish between ambiguous characters, (e.g., w, w; 0, O, 0; 1, 1). 

3.6 Italics are used for emphasis. Indicate italics by underlining on the manuscript. 

4. Figures and Tables 

4.1 All figures and tables should be numbered consecutively with arabic numerals, with 
titles which are as nearly self explanatory as possible, at the bottom for figures and 
at the top for tables. 

4.2 They should be put on separate pages with an indication of their appropriate place- 
ment in the text. (Normally they should appear near where they are first referred to). 

Ss References 

5.1 References in the text should be cited with authors’ names and the date of publication. 
If part of a reference is cited, indicate after the reference, e.g., Cochran (1977, p. 164). 

5.2 The list of references at the end of the manuscript should be arranged alphabetically 


and for the same author chronologically. Distinguish publications of the same author 
in the same year by attaching a, b, c to the year of publication. Journal titles should 
not be abbreviated. Follow the same format used in recent issues. 
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Tableau 1 _ 
Schéma de publication des estimations successives d’un aggrégat* S 
= 
Période de référence re: 
Période de S 
ublication a) 
B Feral to eC hoe “Poem S. eee Peet Se. = ram =. = O72 hae Fi co 
- 
3 
t—2h+2 1Pr—2n41 c 
& 
2P1—2h+1 @ 
i= 
5 
1P1—h at 
ice) 
t—2h+m+2 mPt—2h+1 ae” eg 2P1—h Pr h+1 © 
t—2h+m+3 Vash at ae ie 3 yi reo 
7 1Pt—m 
mPt—h . 2P1—m \P 
Teh mPtHh+1 2P \P 
Tras oP Pe 1 
Meteo - Pi=m —: . 2Pr-1° WP 
t+2 me le ae Py Pris 
Vip. Ty?’ 2Pr4i 
r mPt-1 % 
Ty mPt ; = 1Pr—m 
t+m+2 tom Pie a rem 
t+m+3 T i <a 3 : P. \P. 
b/t—h+1 b/1—m . : birtiee bbls 2 . . oP. Pre 
mPT—m : : Pr 1Pr 
T +2 rem mP. . , 2Pr 1Pr+i 
* * 
T+3 s’t—2h+1 % "i sth st—h+1 ‘ s't—m . , slt-1 st s't+1 5 ; s'T-m i mP ° 2PT +1 


4 Le processus de révision prévoit m révisions successives de type courant (2p, ..., mP,/), une révision aux jalons due aux contrGles effectués toutes les h périodes (pr) 
et une révision extraordinaire a l’occasion de laquelle on reconstruit une série rétrospective a partir du temps T + 2 (Source: Biggeri 1984, p.24). 
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In This Issue 


This issue of Survey Methodology contains papers covering a broad range of topics. In the 
first paper, Alho investigates different estimators of the variance of the population size estimated 
using dual registration. The bias of the usual variance estimator, derived under the assumption 
of homogeneous capture probabilities, is investigated under population heterogeneity and two 
alternative variance estimators are proposed. The three estimators are applied to occupational 
disease data from Finland. 

Rao and Shimizu compare three procedures for combining independent estimates obtained 
at successive time periods. They show that all three procedures produce an improvement over 
the use of the estimate based on only one occasion. The findings are illustrated by applying them 
to data from the American National Health Discharge Survey. 

In their paper, Lavrakas, Settersten and Maier take a descriptive look at the problem of panel 
attrition in surveys, using data from two surveys carried out using random digit dialing. The 
paper gives the reader a good introduction to some of the causes of attrition, with suggestions 
on how its effects can be reduced. 

Sutradhar, Dagum and Solomon give an exact test for the presence of significant stable 
seasonality for time series with seasonal patterns that are stable over time except for possible 
annual shifts. The assumptions of the standard ANOVA F-test used by the X-11-ARIMA sea- 
sonal adjustment method are violated when the residuals are autocorrelated. The exact test, how- 
ever, takes into account the possibility of autocorrelated residuals. The exact and the standard 
tests are compared for several Canadian socioeconomic series. 

A characteristic of quota sampling is the absence of randomized selection. Some kind of modelling 
must be imposed to construct estimators. The traditional approach is to use superpopulation 
modelling, and Deville proposes interesting extensions to this method. An approach suggested 
by the author is to model the sampling process. Comparisons with random sampling are made. 

While household surveys are successfully used to collect data about human populations, they 
are not suitable for studying the characteristics of mobile human populations, such as visitors 
to museums or parks, shoppers, efc. Kalton describes different sample designs for surveys of 
flows of human populations and provides a number of examples of such surveys. The examples 
illustrate that field work considerations play an important role in the choice of a sample design. 

Hidiroglou, Choudhry and Lavallée provide a sampling methodology for continuing sub- 
annual business surveys. A rotation scheme is suggested to maintain a representative sample 
through time. The properties of a number of estimators of totals for this sampling methodology 
have been evaluated in an empirical study which reflects a number of possible survey conditions. 

Stasny, Goel and Rumsey use regression models to obtain small area estimates of wheat pro- 
duction when the data come from non-probabilistic sources. A simulation study compares the 
estimates obtained through this approach with the standard synthetic and direct estimators. Three 
scaling methods to satisfy additivity constraints are also compared. 

In the last paper of this issue, Norris and Paton give an overview of Canada’s five year old 
General Social Survey. They present a brief account of the information needs and discuss the 
five annual topics addressed by the survey. A description of the survey’s methodology and 
experiences with the use of random digit dialing are presented. The authors’ analysis of 
nonresponse rates over the life of the survey has implications for other telephone surveys. 
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Variance Estimation in Dual Registration 
Under Population Heterogeneity 


JUHA M. ALHO! 


ABSTRACT 


-The usual dual system estimator for population size can be severely biased, if there is population 
heterogeneity in the capture probabilities. In this note we investigate the bias of the corresponding variance 
estimator under heterogeneity. We show that the usual estimator is conservative, i.e., it gives too large 
values, if the two registration systems are negatively correlated, uncorrelated, or when the correlation 
is positive, but small. In the case of high positive correlation the usual estimator may yield too low values. 
Two alternative estimators are proposed. One is conservative under arbitrary heterogeneity. The other 
is conservative under Gaussian heterogeneity. The methods are applied to occupational disease data from 
Finland. 


KEY WORDS: Capture-recapture; Dual system; Heterogeneity; Occupational diseases. 


1. INTRODUCTION 


Suppose there are N individuals in a closed population. The problem is to estimate the 
unknown N using dual registration. We sample twice with n; individuals captured at the jth 
time, j = 1, 2. Let m be the number captured twice. Define indicator variables u;; and mj; 
fori = 1, ..., N such that uj; = 1, if and only if individual / is captured at the jth time 
only, 7 = 1, 2; and m; = 1, if and only if individual / is captured twice. Otherwise u,; and m; 
are zero. Define nj; = uj; + m; as the indicator of capture at the jth time, j = 1, 2. Let 
M; = uy; + uy; + m; indicate capture at least once. Define the individual capture probabil- 
ities as pj, = Elnj),7 = 1, 2; and pj; = E[m;,]. Assume that the probabilities are strictly 
between zero and one. The fact that the probabilities are allowed to vary by individual indicates 
that we may have population heterogeneity in the capture probabilities. We complete the defini- 
tion of the dual registration (or capture-recapture) model by assuming that the captures are 
independent for each individual, or p;2; = p1;P2;, and that the multinomial vectors 


(Uyj, U;, Mj 1 — Mj) ~ Mult; 24;92;, P2911 PurP2i. 1 — )); 


where gj, = 1 — pj, j = 1, 2, and $; = Pi; + Pri — Pri, are independent fori = 1, ..., N. 

It is well-known that when capture probabilities do not vary by individual, or pj; = p;, 
j = 1, 2, the maximum likelihood estimator of N is N = n,n,/m (or more precisely, the 
largest integer short of this value; cf., Feller 1968, p. 46). This classical estimator can be severely 
biased under population heterogeneity (Seber 1982, p. 565; Burnham and Overton 1979, 
Table 4, pp. 931-932). As shown, e.g., in Example 1 below, under homogeneous capture 
probabilities the asymptotic variance of Nis Var(N) = Nqiq2/(p1p2), where g; = 1 — pj, 
j = 1, 2. Then Var(N) can be estimated by V; = n,n .u,u,/m? (Sekar and Deming 1949, 
pp. 114-115). 


! Juha M. Alho, Institute for Environmental Studies and Department of Statistics, University of Illinois at 
Urbana-Champaign, 1101 W. Peabody Dr., Urbana IL, 61801, U.S.A. 


124 Alho: Variance Estimation in Dual Registration 


The purpose of this note is to investigate the adequacy of the variance estimator V,, and 
compare the bias of V, to the bias of N. One motive for investigating V, is that it has not been 
previously known whether V, is adequate in the case in which there is population hetero- 
geneity, but Nis, nevertheless, consistent. This turns out to be the case. Similarly, it has not 
been clear when V, gives overestimates and thus can lead to valid confidence intervals, despite 
the bias of N. This turns out to be possible for one-sided intervals in special circumstances. 

In Section 2 we calculate the asymptotic variance of N, as N — 0, and derive a conservative 
estimator V, for this variance under arbitrary heterogeneity. In other words, V, overestimates 
the true asymptotic variance. One might hope that an overestimate of variance could compen- 
sate for the typically negative bias of Nand still yield valid confidence intervals. Unfortunately, 
this appears possible only when the bias of N is small, or when N is small. In Section 3 the 
adequacy of V, is studied under Gaussian heterogeneity and an estimator V; is derived, which 
is conservative under this restricted type of heterogeneity. Gaussianity per se is not required 
for the arguments, only that the moments of the pairs (p);, P2;) agree with those of a bivariate 
Gaussian distribution. This setup permits the ready examination of the effect of correlation 
between p;;’s and p>;’s on variance estimation, because correlation is expressible in terms of 
just one parameter, the ordinary moment correlation coefficient. In Section 4 we compare the 
bias in variance estimates to the bias of N using empirical data relating to the registration of 
occupational diseases in Finland. 


2. BIAS AND VARIANCE UNDER HETEROGENEITY 


Define fj, as the average probability of capture at the jth time, j = 1, 2; and let D2, be 
the average of the products p;po;, i = 1, ..., N. Then, Cy = Dyn — PinDon is the 
covariance of the pairs (pj), P2;). Assume that the limits Diy — pj, 7 = 1, 2; Dizw > Dio, and 
Cy — C exist. Then we have that N/N — p,p>/p,,so N/N — 1 — — C/py,asN — o. 
This is the asymptotic bias of the classical estimator under population heterogeneity. Interestingly, 
it only depends on the first two moments of the distribution of the pairs (pj), P2;). As is 
well-known (Sekar and Deming 1949, pp. 105-106; Seber 1982, p. 86), when the covariance 
is zero (C = 0), then the classical estimator is consistent; if C > 0, N gives an underestimate; 
and if C < 0, it gives an overestimate. As noted above the adequacy of V,, when the p,;’s vary 
from one individual to the next but still C = 0, is of particular interest. 

We shall now calculate the asymptotic variance of the classical estimator under our general 
heterogeneity model. Note that the finite variance does not exist, because there is a positive 
probability that m = 0. Therefore, ‘‘asymptotic variance’’ properly refers here to the variance 
of the limiting distribution rather than to limit of the variances, as N — ©. 


Lemma 1. The asymptotic variance of N is 


m2 a2 p ey. a) > 
Var(N) = Nios phe PIED P25 ee i 5, 


where S; = S;/N for j = 1, ..., 5, with 
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The proof is sketched in the Appendix. We note that unlike the bias of N that depends on the 
first two moments of the pairs (pj;, P;) only, Var(N) depends on moments up to fourth 
order. In special cases, such as the ones considered in Example 2 and Proposition 2, a simpler 
representation is possible. 

Example 1. Suppose there is no heterogeneity in the probabilities, or p;; = p,;,j = 1, 2. Then 
By = Py J = 1, 2; Bio = Piprs S; = pj, j = 1,2; 53 = pip3, Sy = pipr, and Ss = p,p3. 
Hence, the asymptotic variance is Var(N) = N(1 — py — Po + Pi P2)/(PiP2) = NqQ1qQ>/ 
(p\P2). Consistent estimators for Np;p, and Np; are m and n,, j = 1, 2. In other words, 
Npj/nj — 1,7 = 1, 2, and Np, p/m — 1, as N — o. This gives us V; as an estimator for 
Var(N). 

Example 2. Suppose that the pairs (p;;, P2;), i =1, ..., N, are independent in the sense that 
the distribution of p,;’s is the same for each distinct value of the p,;’s. Then, f,. = p,po, 
S; = S,55, S, = p.S,, 5; = p,S,. Substituting into the Lemma we get 


s 1 1 1 5S S Ss 
PHOS eae ee Se 
PiP2 P2 P\ Pike... Pi P2 
14 
aly ae — cv(p,j)” ev(pa)) ; 
1P2 


where cv(pji) = (S; - Pi) /p;, is the coefficient of variation of the p;’s, 7 = 1, 2. Obviously, 
Var(N) < NG,G2/(,p.). A comparison with Example 1 shows that V, is a conservative 
estimator of Var(N) (i.e., V; is asymptotically too large), when p,;’s are independent of p>;’s. 
Another way of saying this is that, given the means p;, j = 1, 2, the largest value of the 
variance is obtained at homogeneity. This is analogous to the variance of the number of suc- 
cesses in Bernoulli trials with variable probabilities of success, cf. Feller 1968, pp. 230-231. 
A comparison with Example 1 shows that V, is a conservative estimator of Var(N) (i.e., V, 
is asymptotically too large), when the pairs (pj;, P2;) are independent. Note that the independ- 
ence condition implies that C = 0. 

When the probabilities are not independent, the classical variance estimator is not guaranteed 
to be conservative. A conservative estimator exists, however. It is obtained by majorizing 
Var (N) by a quantity that can be estimated in terms of the observable variables. We prove 
in the Appendix the following general proposition. 


Proposition 1. A conservative estimator of Var(N) is 
Vy = (nink + nomu, + nimu)/m>, 


where uj = nj — m,j = 1, 2. 
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3. GAUSSIAN HETEROGENEITY 


We shall now turn to a special case in which the sample moments of the pairs (p;, P2;), 

i = 1, ..., N, agree with those of a bivariate normal, or Gaussian, distribution. This will 

permit a much sharper specification of a conservative variance estimator than the one obtained 
in the general case above. Assume that 

XxX, My ViMT Vi V2 py ho 

Ny ‘ 

X, M2 PV1V2 Hy M2 Veni 


where |p| < 1, andO < yp; < 1,/ = 1, 2. Note that v,’s can be interpreted as the coeffi- 
cients of variation of the distributions of p;;’s. Write S; = S;/Nforj = 1, ..., 5, as before. 
Then substitute the moments of the bivariate normal distribution into Lemma 1 as follows, 
Bp; = E(X;) = wy, J = 1, 2; 
S; = ELXJ] = wf + v7), 7 = 1, 23 
Pin = E[X,X2) = wyp2(1 + viv); 


S; = E[X{X3] = wjw3(1 + vi + v3 + 4pvjvy + (2p? + 1)viv3); 


= wipe(1 + 2pvyv. + vi); 


i 
T 
Ls 
sy 
S 
| 


Ss = ELX,X3] = wyy5(1 + 2pvjv2 + v3). 


Straightforward, but slightly tedious calculations prove then the following proposition (details 
omitted). 


Proposition 2. With the above assumptions 
Var(N) = A,A, + RN, 
where 


A, = N/Q“#pviv5)7: 


Ay = [1 — (1 + wo) + pvyyn + poe + 0v1¥2)71/ [ei p2(1 + pvyv2)7); 


R = (2pv,v2 + 3p?vi v3 — p*vivs — p?vivg — viv3)/(1 + pyyv2)*. 


We can evaluate the classical variance estimator V; = n,n u,u>/m? using this result. Note 
first that {,m./m}/A, — 1, as N — oo. Similarly, {u,;u,/m7}/A, — 1. This proves the 
following corollary to Proposition 2: (V, — Var(N))/N — —R, as N — o. For example, 
if p = 0, then —R = viv3, so that V, is seen to overestimate the asymptotic variance. This 
is in accordance with Example 2. 
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How reasonable is the assumption of Gaussian moments? Certainly the capture probabilities 
cannot have strictly Gaussian distributions, because the Gaussian distribution always puts some 
probability mass outside the unit interval. On the other hand, suppose we generate the p;;’s 
by taking logit(p;;) = a; + b;Y¥j;, where the pairs (Yj;, Y;) are a sample from a bivariate 
normal distribution with mean zero, unit variances, and correlation p. If we have the relations 
a; =-logit(u;),7 = 1, 2,andb; = v,(1 + Hj), then the assumption of Gaussian moments 
is approximately true. In fact, even the distribution of the pairs (p;, p;) is in that case 
approximately bivariate Gaussian. 

Let us consider the adequacy of V,; further, under the Gaussian moments. The fact that pro- 
babilities are constrained to be between 0 and 1 means that y;’s are between zero and one. 
Moreover, to be sure that most of the probability mass is in the unit square, let us assume that 
0<v s %,j = 1, 2. Ify,;’s are close to one, a much smaller upper bound would be needed. 
Assume now that p s 0. Then, one can show that 


—R => (p*viv3 + p*viv3)/(1 + pviv2)* > 0, 


so that V, overestimates Var(N) for p < O also. Note that by continuity V, must overestimate 
Var(N) for some positive values of p, as well. 

One can show that R = R(p) is an increasing function of o for at least o > 0. In the limit 
we have 


—R(p) — (-— 2vy,y, - Day ANG + Vive + vive)/(1 + Ree es 


asp — 1. When 0 < vy; Ss %, / = 1, 2, the smallest value of the above limit occurs at 
Vv; = v2 = %. The minimum value is — 152/625 > —1/4. Consequently, foro > 0, V; can 
either underestimate or overestimate Var(N). 

The practical implications of the above results are as follows. First, if < 0, then Nis either 
consistent or it overrestimates N and V, gives an overestimate of the variance, so we can 
calculate a conservative upper confidence limit for N. When p > 0, N gives an underestimate 
of N. If, in addition, p is small, then V, gives an overestimate, and we can get a conservative 
lower confidence limit for VN. Obviously, these are rather special circumstances that one would 
not expect to be of wide practical utility. 

Under the present model the asymptotic bias of V; is > —N/4 for all values of o. We can 
derive a conservative variance estimator by noting that in the Gaussian case the asymptotic 
relative bias of Nis —pv,v2/(1 + pv, v2) = —1/5. Hence, asymptotically 5N/4 => N. A 
conservative estimator of Var(N) is, for example, V; = V; + 5N/16. This can be much 
smaller than V, indicating that the Gaussian assumption is a very powerful one. 


4. AN APPLICATION TO OCCUPATIONAL DISEASE 
REGISTRATION DATA 


To get an idea of how large the biases may be in practice, let us look at occupational disease 
data from Finland as an example. The Finnish Register of Occupational Diseases has been in 
operation since 1964. It is kept by the Institute of Occupational Health in Helsinki. Since 1975 
the number of new cases reported to the Register has varied from about 4,000 to over 7,000 
annually (0.2 - 0.4 % of the employed population). Noise-induced hearing loss, diseases caused 
by repetitive of monotonous work (epicondylitis, bursitis, tendinivaginitis), and skin diseases 
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are the major diagnostic groups (cf. Vaaranen et a/. 1985). The Register can be viewed as a 
dual registration system, because each case of disease should, under existing regulations, be 
reported to the Register both from the appropriate insurance company and the examining 
physician. 

It is likely that the probability of reporting a case depends on diagnosis, for example. Indeed, 
based on data from the year 1981 we get the following statistics. Reports from the insurance 
companies, 1, = 3,769; reports from the physicians, nm. = 3,053; and cases reported from 
both sources, m = 1,591. Thus the usual dual registration estimate is N = 7,232 with 
Vi= 97, Vi= 222, and V;?= 108.0. The closeness of V; to V, is striking. Stratifying the data 
into four categories by diagnosis (the three diagnostic groups mentioned above, and the 
remaining ‘‘other’’ category) yields the ee estimates. Noise-induced hearing loss: 


N = 2,230, V/?= 33.4, V;*= 47.2, and V;?= 42.6; diseases caused by repetitive or monot- 
onous work: N = 3,572, ine ; 201. 4, Via 303.8, and V;*= 204.2; skin diseases: N = 1,441, 
Vii= 30.9, V/= 86.2, and V;*= 37.5; other diseases N = 1,015, Vix 32.7, V-= 79.1, and 


Y= 37.2. Adding the abies yields the following estimates for the total number of diseases: 
= 8,258, V;*= 209.0, V3*= 340.3, and V;*= 215.2. We see that diseases caused by 
eae or ela eae are ee to a particularly great extent. 

The analysis was extended further by stratifying the data by diagnosis (4 categories), 
insurance company (11 categories), and main groups of industry (7 categories). A priori, these 
factors could be thought to have an influence on reporting probabilities. However, the stratifica- 
tion did not alter the point estimate materially. It did increase the estimated standard devia- 
tions by over a third, apparently because some of the strata became very small. We conclude 
that the bias in the point estimator caused by diagnosis is the dominant source of error in the 
classical estimator in this application. 

The same data were further analyzed using a logistic regression technique that allows us 
to take into account observable population heterogeneity due to both discrete and continuous 
explanatory variables. In this application age was shown to have an effect on reporting pro- 
babilities within the diagnostic groups for one source of information, but not for the other. 
Therefore, the point estimates remained unchanged and the conclusion regarding the role of 
diagnosis could not be refuted (Alho 1990). 


5. DISCUSSION 


Our theoretical results indicate that the usual variance estimator V, is conservative when 
the two registration systems are negatively correlated or independent. By continuity the 
estimator may be conservative also when the correlation is positive but small. Under high 
positive correlation V; gives too low values. We introduced an alternative estimator V,, which 
is conservative under arbitrary population heterogeneity. However, it appears to be unduly 
conservative in view of the numerical comparisons with V3, which is guaranteed to be conser- 
vative under Gaussian heterogeneity. The closeness of V3 to V; suggests that, in practice, V, 
may be fairly robust against population heterogeneity. 

Unfortunately, even the use of the conservative estimator V, would not have been sufficient 
to cover the bias in the classical point estimator in our empirical example. Perhaps this was 
to be expected, since the bias of N and the degree of overestimation provided by V; are both 
of order N. Hence, the use of V, inflates the width of a confidence interval by a factor of order 
N* only. Therefore, V; can compensate for the bias of N, if the bias is small, or if N itself 
is small. Hence, it seems that the successfull application of the dual registration method requires 
that either we have roughly uncorrelated registration systems, or that the heterogeneity is 
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observable. In the latter case we may use stratification as suggested already by Sekar and Deming 
(1949), or logistic regression modeling as suggested by Huggins (1989) and Alho (1990), to adjust 
for the bias of the classical estimator of population size. 
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APPENDIX 


Proof of Lemma 1. Apply a linear Taylor-series development to N = n,n>/m at E[n,] 
E[n2)/E(m] = Np, p2/py2, or 


Pip 
3 gee (nig NAY ee (ip NDs) fe 
P12 P12 Pi2 Pir 


(in — NP j>): 
Hence, we have 
a Nipime ne a. \ 2 5 \2 5. A.\ 2 
e| (8 = a |- ey Var(m) + (7) Var(n) + ica Var (m) 
P12 P12 P12 P12 


_ PBR Cov(n,m) — PiPz 
Pi2 Pio 


Cov(n2,m). 


Under our independence assumptions Var(n;) = Np; — S;,j = 1, 2; Var(m) = Npy — $3, 
Cov(n,,m) = — Sy + Np, and Cov(n,,m) = — Ss; + Np. Substituting these into the 
mean squared error gives the result. 


Proof of Proposition 1. We ignore the negative term containing S$; in Lemma 1. Since 
0 < pj < 1, we have Ss < Nfj2, and S, < S,. Therefore, 


27, D2 ae ao) 2 
PP? 5 Pia ng 4 P25 4 mee a) P2 np), 
Pp P12 


Similarly, 


Substituting these bounds to the expression of Lemma | we get 


ots ese = _ = =) = _ = 
Pips N+ (D, Pi2)P2 np 4 (D2 — Pi2)P 


Var (N) < 25 . 
Pip P12 P12 


Estimating Np; by n;, 7 = 1, 2; and Np;2 by m we get the result. 
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Combining Estimates from Surveys 


PODURI S.R.S. RAO and I.M. SHIMIZU! 


ABSTRACT 


For estimating the proportion and total of an item for the present occasion, independent estimates at 
the current and previous occasions are combined through three different procedures. In the first one, 
trend over the occasions is utilized. For the second one, the One-Way Random Effects Model is employed. 
The third procedure uses the Empirical Bayes approach. All the three procedures are seen to perform 
better than the sample estimates obtained from the data of the current occasion alone. Advantages of 
these methods and their limitations are discussed. All the procedures are illustrated with the data from 
the National Health Discharge Survey. 


KEY WORDS: Trend; Weighted least squares; Random effects; Improved estimation; Biases; Mean 
square errors. 


1. INTRODUCTION 


In several national surveys, independent samples are obtained at successive time periods. 
In this paper, information from the past surveys is utilized to improve the estimates for the 
current period. For the sake of illustration, we consider the National Health Discharge Survey 
(NHDS) in the U.S. In this survey, which has been recently redesigned, a three stage sampling 
design is used with geographical regions as the Primary Sampling Units (PSU’s) at the first 
stage. Hospitals and discharges are selected at the second and third stages respectively. The 
survey collects information on various items of the patients like age, sex, racial characteristics, 
length of stay, diagnosis, and surgical and non-surgical procedures. The selected PSU’s and 
hospitals remain in the study for a certain number of years. Independent samples of discharges 
are obtained every year from the selected hospitals. Shimizu (1987) presents further details of 
the redesign of the NHDS. 

At present, for a given hospital, estimates of the proportions for the different items for the 
current year are obtained only from the data of this year. National estimates are obtained by 
suitably weighting these proportions with the reciprocals of the probabilities of selection of 
the hospitals and the PSU’s. However, Bean (1987) found that for most of the items the 
estimates are somewhat correlated over the years. For the sake of illustration, sample 
proportions obtained from the NHDS for 1977-86 for Acute Myocardial Infraction (AMI) and 
Mental Disorders (MDS) are presented in Table 1 for three hospitals and they are exhibited 
in Figures 1 and 2. Examination of the proportions for these three and 17 more hospitals 
suggested that the inclusion of past information can increase the precision of the estimates for 
the current year. 

It should be cautioned that the sample proportions in Table 1 or Figures 1 and 2 should 
not be used to make inferences regarding the increase or decrease of AMI or MDS in the entire 
population. 


! Poduri S.R.S Rao, Department of Statistics, Hylan 703, University of Rochester, Rochester NY,14618 U.S.A., and 
I.M. Shimizu, National Center for Health Statistics, Office of Research and Methodology 1-68, 3700 East-West 
Highway, Hyattsville MD, 20782, U.S.A. 
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Table 1 


Data from the National Health Discharge Survey for 1977-86 
Sample totals and proportions for Acute Myocardial 
Infraction (AMI) and Mental Disorders 
(MDS) for three hospitals 


NOvot ae AMI MDS 
Year pve es acne a — Sample athe Sample 
n proportion proportion 
1977 9,416 276 5 .018 37 A: 
1978 10,234 266 7 .026 24 .09 
1979 9,354 294 9 .031 39 Sie! 
1980 10,372 S20 9 .028 41 me 
1981 10,712 342 8 .023 45 urs 
1982 10,683 309 9 .029 43 14 
1983 10,935 360 7 .019 46 fa 
1984 10,090 330 6 .018 50 af 
1985 10,431 297 8 (O27 41 14 
1986 10,247 264 4 .015 35 a 
1977 6,720 474 9 .019 18 .04 
1978 6,710 470 14 .030 25 05 
1979 6,970 495 8 .016 28 .06 
1980 6,794 466 14 .030 29 .06 
1981 7,055 486 9 .019 34 .07 
1982 6,265 442 9 .020 24 .05 
1983 6,234 442 10 .023 28 .06 
1984 6,221 439 9 021 15 .03 
1985 6,063 375 8 .021 19 05S 
1986 5,781 71 4 O11 12 .03 
1977 6,400 606 on .0347 41 .0677 
1978 6,286 635 23 .0362 42 .0661 
1979 6,494 554 12 .0217 27 .0487 
1980 6,813 571 7 .0298 2 .0438 
1981 7,430 729 14 .0192 32 .0439 
1982 7.26). 712 20 0281 39 .0548 
1983 7,110 694 23 .0331 43 .0620 
1984 7,268 718 a5 .0487 29 .0404 
1985 6,716 657 19 .0289 45 .0685 


1986 6,464 655 21 .0321 33 0504 


In this article, we examine three procedures for improving the estimates for a specified 
hospital by utilizing the information from the current and the previous years. In the first 
method, estimates of the proportions are obtained from the linear trend over the years and 
the Weighted Least Squares Method. If there is a significant positive or negative trend over 
the years, this method will have higher precision than the sample estimate of the current period. 
If the trend is not pronounced, the increase in precision will be negligible, as expected. 

For the second procedure, the One-Way Random Effects Model with unequal variances is 
used to combine the information. Yates and Cochran (1938) and Cochran (1954) suggested 
this type of procedure for combining information from experiments conducted at different 
time periods and locations. While the Analysis of Variance (ANOVA) method had been used 
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Figure 1. Proportions for AMI: 1977-86 
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Figure 2. Proportions for MDS: 1977-86 
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for quite some time for this purpose, C.R. Rao (1970) suggested the Minimum Norm Quadratic 
Unbiased Estimation (MINQUE) and demonstrated its advantages. P.S.R.S. Rao, Kaplan and 
Cochran (1981) examined the relative merits of the ANOVA, MINQUE and several related 
procedures. We have employed the estimation procedures related to these methods. The estimate 
for the proportion obtained by any of these procedures is a weighted combination of the 
estimates of the different time periods. The weights depend on both the between and within 
variances of the time periods. In the third procedure, the Empirical Bayes approach is used 
to estimate the proportions for the current period. 

We denote the above three procedures by TR, VC and EB respectively. The notation is 
presented in Section 2. The sample estimator for the proportion and its variance are given in 
Section 3. The above three estimation procedures along with the expressions for their Standard 
Errors (S.E.’s) are presented in Sections 4, 5 and 6. We have used these expressions to compute 
for 1986 the sample proportions, the above three types of estimates, and their S.E.’s for 20 
hospitals in the NHDS. These estimates for the three hospitals mentioned earlier are presented 
in Table 2 for AMI and Table 3 for MDS. Results from the entire study are described in 
Section 7. The final section contains a discussion of the results and topics for further research. 


Table 2 


Estimates of the Proportions for 1986 and S.E.’s (bottom figures) 
for Acute Myocardial Infractions (AMI) 


Variance 


Hospital Sample eat components a 
P proportion estimate ae estimate 

1 .0152 .0196 .0224 .0224 

.0070 .0046 .0026 .0003 

p: .0108 .0162 .0204 .0203 

.0048 .0036 .0031 .0003 

3 .0321 .0319 .0304 .0309 

.0060 .0038 .0028 .0037 

Table 3 
Estimates of the Proportions for 1986 and S.E.’s (bottom figures) 
for Mental Disorders (MDS) 
Variance 

, Sample Trend Bayes 
Hospital proportion estimate re et estimate 

1 .1326 1431 .1292 .1292 

.0205 .0115 .0060 .0010 

2 .0323 .0437 .0500 .0427 

.0087 .0056 .0039 .0057 

3 .0504 .0496 .0534 .0523 


.0080 .0049 .0032 .0048 
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It should be mentioned that for the problem considered in this paper, the samples are drawn 
independently at the different time periods. Secondly, the population proportions for the 
previous periods are not known. Because of these reasons, the usual ratio and regression 
methods cannot be employed to improve the accuracy of the estimators for the current period. 
For the same reasons, the estimation procedures suggested in the literature for the rotation 
sampling schemes cannot be used in this situation. In spite of these difficulties, the three methods 
considered in this paper can be used to estimate the population quantities with a high accuracy. 
When summary figures at the different periods are available, public and private users can obtain 
these estimates and their standard errors without much difficulty. These procedures can also 
be used when there is nonresponse during some years - some of the hospitals do not provide 
information to the survey during some years. 


2. NOTATION 


We present in this section the notation for a selected PSU. Let y,; denote the jth observation 
on the sampled discharge on an item like the number of surgical cases at time¢ = (1, 2, ..., 7), 
from the ith hospital, i = (1,2, ..., K), which has N, discharges. Note that K may change 
over the years due to nonresponse or the addition of new hospitals. 

The total and mean at time ¢ are 


and 


Yir = Yir/Nit- (2) 


The total and mean of the sample of size n;, from the N,, discharges are 


Nit 
Jin sD Sitj (3) 
1 
and 
Vit = Vit/Nit- (4) 


To estimate the total number and proportion for a specified item, let y,,; = 1 if the obser- 
vation belongs to that item, and zero otherwise. With this notation, the total and proportion 
for an item at time ¢ can be written as A;, and P, = Aj;/N;. Note that P;, is the same as Yj, 

In the following four sections, for the sake of convenience, we suppress the subscript / and 
describe the estimators for a given hospital. 


3. SAMPLE PROPORTION 
An unbiased estimator of the proportion P, for an item like AMI or MDS is 


P, = a,/n,, (5) 
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where a, is the number of cases of that item observed in the n, sample discharges. The variance 
of P, and its unbiased estimator are 


4 N, — ny Plo F) 
AONE ic a ea (6) 
t t 


and 


= Pil =sp 
(Bye hope ye aly (7) 
n,— 1 


where f, = n,/N,. Note that P, is the same as p, = ¥ Vy/Nt- 


4. LINEAR TREND 


The sample observations y,, j/ = (1, 2, ..., mj) can be written as 
Vij = be + Ey, (8) 


where p, is the mean for the ith hospital at the fth period, and e¢,; is the random error with 
expectation zero and variance 07 = P,(1 — P,). Since the samples are drawn independently 
during each year, the errors e,; are uncorrelated from one year to another. 

With the assumption of a linear trend, the sample mean can be expressed as 


J, =a + Px; + &, (9) 


where x, = tandé, = ) f‘e,;/n;. Further, V(é,) = (N; — n,)o7/(N, — 1)n; =1/W,. Note 
that with the zero-one notation, j, is the same as P,. The WLS estimators of 8 and a are 


5 LWAx% — 2); 


g = : (10) 
Le = x)" 
and 
&=y — Bx, (11) 
where xX os p23, W,x,/¥ W, and y = y W.9,/¥ W,. 
Estimator of p, is 
b= a + Bx, 
= 9 +h(x%, — x). (12) 


This is the Trend Estimator (TR). Estimators of this type for infinite populations have been 
examined in the literature; see Carroll and Rupert (1988), for instance. 
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We have obtained the estimate of yp, from this expression by replacing W, with 
w, = (1 — fy) &?/n,, where 6? = P(1 — B). If it can be assumed that for large N, the dis- 
tribution of y,; is normal, y, will be independent of w,. In this case, the expression in (12) 
remains unbiased for y,. Even if the assumption of normality is not valid, it can be seen that 
w, approaches W, for large n, and hence the expression in (12) with the estimated weights 
approaches py. 


The variance of the above estimator is 


p= a 


Vii.) = —— + — 4... 
MW, DW, — %)? 


(13) 


We have estimated this variance by replacing W, by w,. The bias in the resulting estimator will 
be small for large 7;. 

For the illustration in this article, ¢ = (1,2, ..., 10), that is, 7 = 10. For 1986, we have 
found the estimate for the proportion of an item and its S.E. from (12) and (13) with x, = 10. 


5. VARIANCE COMPONENTS MODEL 


Examination of the proportions for the AMI and MDS of the 20 hospitals for the ten years 
showed no specific linear or nonlinear trend. For all of them the patterns somewhat resembled 
those of the three hospitals, presented in Figures 1 and 2. These observations indicated that 
the proportion for AMI or MDS for the current year can be obtained by combining the infor- 
mation from all the ten years. The One-Way Random Effects Model can be used for this purpose. 


The model in (8) can be written as 
Vip = w+ (Me — wd) + ey 
= Pp Oo a5 Ej. (14) 


If 4, is considered to be randomly drawn from a population with mean yp, the random effect 
a, will have mean zero and variance o2. It is assumed to be independent of €,;. The sample 
mean (proportion) can now be written as 


YrH= uta, t+ &, (15) 
where é,; has mean zero and variance (1 — f;) o2/N;. Thus, from (15), 
ze 2 2 | 
V(¥,) = og + (N — My )oV/(N; — 1); = ie (16) 


t 


The WLS estimator of p is 


(17) 
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This is the Variance Components Estimator (VC) and its variance is 


Vn) = 17), U,. (18) 


For obtaining the mean in (17) and its variance in (18), we have replaced a? by its estimate 
P.(1 — B). Procedures like the ANOVA and MINQUE are available for estimating 02. The 
MINQUE depends on the a priori values r, of (07/02). A related procedure called the 
Unweighted Sums of Squares (USS) method does not depend on 7; and it is described below. 
P.S.R.S. Rao, Kaplan and Cochran (1981) found that this method provides estimates for o2 
comparable to the ANOVA and MINQUE, unless 7, or r; is very small. The USS is computa- 
tionally less cumbersome than the MINQUE. With y* = ()¥,)/T, from (15), 


ELLY (% — y*)7] = (T - 1)02 + (T - 1)(¥yv,)/T, (19) 


where v, = (N, — n,)P,(1 — P,)/(N; — 1)n,. The USS estimator for o2 is 
Gat Syn Pema Bids NUE sal irl alll (20) 


where f, = (1 — f,) B(1 — B)/(n, — 1). If Nis large relative to n,, the sampling fraction 
f, can be set to zero. We have estimated U, from (16) by estimating o2 from (20) and the second 
term by ¥,. Utilizing this estimate of U,, we have estimated yp from (17) and its variance from 
(18). If o2 is much larger than v,, the estimator fi in (17) will be close to ¥*. In this case, estima- 
tion of U; as described above can be expected to have almost no effect on fi. Since fj depends 
only on the relative values of U,, this conclusion can be expected to be valid even when o? is 
not considerably larger than v,. Thus, estimation of U; can be expected to result in only a 
negligible bias for j. 

As is well-known, all the procedures for estimating o2 unbiasedly can result in negative 
estimates. In such a case, we have employed the usual practice of substituting a small positive 
quantity for the negative estimate. In Rao et a/. (1981) it was found that unless o2 is very small, 
this adjustment results in only a negligible bias for 62 and an insignificant increase in its stan- 
dard error. Further, unless o2 is small, the difference in the MSE of fi for the USS and other 
methods of estimating U; was found to be negligible. 


6. BAYES’ ESTIMATOR 
The discussion in the beginning of Section (5) suggests that u, can be assumed to have a 
prior distribution with mean p and variance o2. With the assumptions that for large N, the dis- 


tribution of y,; is normal with mean py; and variance o?, and that the prior distribution of p, 
is also normal, the Bayes’ Estimator for p, is 


B, = E(m,| ¥;) = (1 — a) yy, + ayy, (21) 


where a, = v;/(o2 + v;). The expression for y, is the same as given in the previous section. 
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For given j,, the variance of the above estimator is 


1 


V(B,) = ———_.. 22 
' * G/je) +e py) oe 

With estimates 62, 6? and ji, the expression in (21) can be written as 
B, = (1 — &) ¥, + ap, (23) 


where @, = #,/(62 + %,). This estimator may be called the Empirical Bayes’ estimator (EB). 
Note that f is obtained from (17) with 62 and #,. The variance of this estimator may be 
obtained from (22) by replacing o2 and v, with their estimates. For obtaining the EB and its 
variance, we have estimated o2 and v, from the USS procedure described in the previous 
section. 


7. PERFORMANCE OF THE ESTIMATORS 


We have computed the estimates of P, for 1986 for the 20 hospitals through the different 
procedures described in the previous sections. Since the population values of P, are not known, 
as described earlier, we have found the S.E.’s for the different procedures by substituting the 
sample proportion P, in the place of P,. Since the sample sizes n, are not small, the resulting 
biases in estimating the variances or S.E.’s of the estimators can be expected to be small. 

For the three hospitals, the estimates of P, and the S.E.’s of the different procedures are 
presented in Tables 2 and 3 for AMI and MDS respectively. 

As can be seen from these tables, S.E.’s of TR, VC and EB are smaller than the S.E. of 
the sample proportion. As expected, utilizing the data from the previous periods has helped 
reduce the S.E. of the estimate for the current period. 

Both VC and EB have smaller S.E.’s than TR. However, TR does not require the estima- 
tion of 02. We have found the S.E. of TR to be usually less than 50 percent of the sample 
proportion. 

The EB has smaller S.E. than VC, as expected. Note that VC estimates the overall propor- 
tion, whereas EB estimates the proportion of the conditional distribution. The S.E. of the EB 
becomes close to that of the sample proportion if the sample size is large. 

It is interesting to observe from Tables 2 and 3 that for both AMI and MDS the difference 
between the VC and EB estimates is negligible. The reason for this result is that @, is close to 
unity, which indicates that o2 is small relative to v,. 

The estimates for the total number of cases for 1986 and their S.E.’s can be obtained by 
multiplying the estimates of the proportions in Tables 2 and 3 by the corresponding number 
of discharges N; given in Table 1. 


8. DISCUSSION 


As described in the above section, the results of this investigation recommend the TR, VC 
or EB methods for estimating the proportions and totals for the current period. 

For estimating the S.E.’s of the different procedures, we have utilized the sample propor- 
tions. Further investigation is needed to examine the biases and MSE’s of these S.E’s. 
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For estimating o2 and v,, we have employed the USS. The effects of the ANOVA and the 
MINQUE procedures for this purpose can also be examined. However, the investigation in 
Rao et al. (1981) showed that different procedures of estimating 02 may not have a significant 
effect on the estimation of p or its S.E. 

Further investigation is needed to determine the effect of the different procedures of 
estimating the variances on the EB for p;. 

We have substituted a small positive quantity for a negative estimate of 02. As can be seen, 
this adjustment may result in a small S.E. for both the VC and EB, and may present too 
optimistic a view about the estimates of » and y,;. Further examination of this problem is 
needed. 

We have assumed a linear model for the proportion. The logit or probit transformation can 
be used before using this model. However, large population and sample sizes are needed to 
justify the estimates that can be obtained through these transformations. The estimates pro- 
posed in this article can be obtained by the public and private users by using any simple com- 
puter program. 

Improved estimates for each hospital are considered in this paper. The national estimates 
for a given item like AMI or MDS can be obtained by suitably weighting the above estimates 
by the reciprocals of the probabilities with which the hospitals were selected. Such a procedure 
is expected to improve the precision of the national estimates. 

Time series methods like the ARIMA can be used as suggested for instance by Blight and 
Scott (1973) and Scott and Smith (1977) for estimating the proportions and total numbers. These 
methods will result in different models for different items. Secondly, the available package 
programs for these approaches assume large population sizes and equal error variances, and 
the same sample sizes for all the time periods. Such assumptions are not satisfied for the problem 
we have considered in this article. As mentioned in Section 1, the TR, VC and EB methods can 
also be used when there is nonresponse during some years. 
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RDD Panel Attrition in Two Local Area Surveys 


PAUL J. LAVRAKAS, RICHARD A. SETTERSTEN, Jr. 
and RICHARD A. MAIER, Jr.1! 


ABSTRACT 


This paper compares the magnitude and nature of attrition in two separate RDD panel surveys conducted 
in the City of Chicago (i.e. the surveys were independent studies and were not conducted as part of a 
planned experiment), each with a between-wave lag of approximately one year. For each survey, sampling 
at Wave | was performed via one-stage (i.e. simple) random-digit dialing. In Study 1, respondents’ names 
were not elicited; thus, when telephone calls were made at Wave 2 of Study 1 interviewers could not ask 
for respondents by name. Instead, interviewers asked for respondents by using a gender-age identifier. 
In Study 2, respondent name identifiers were gathered during Wave 1 and were used in Wave 2 re-contact 
attempts. The magnitude of the attrition in Study 1 (i.e. the proportion of Wave 1 respondents not re- 
interviewed at Wave 2) was 47%, whereas in Study 2 it was 43%: a marginal difference in attrition rates. 
In both surveys, age, race, education and income were significantly related to attrition. Discussion is 
presented on the trade-off between minimizing attrition vs. minimizing respondent reactivity as poten- 
tial sources of total survey error. Suggestions for decreasing the size of attrition in RDD panel surveys 
are discussed. 


KEY WORDS: Panel attrition; Random-digit dialing; Telephone surveys. 


1. INTRODUCTION AND LITERATURE REVIEW 


For the past several decades the problem of panel attrition has received only passing atten- 
tion in the survey methods literature. Published articles either have addressed techniques which 
can be employed to minimize the size of the attrition from panel studies (e.g. Droege and 
Crambert 1965; Crider, Willets and Bealer 1971; McAllister, Goe and Bulter 1973; Freedman, 
Thornton and Camburn 1980; and Burgess 1989) or have addressed statistical techniques that 
may be used to adjust for the effects of panel attrition (e.g. Lehnen and Koch 1974; Hausman 
and Wise 1979; Winer 1983; and Lepkowski 1989). 

Few articles have reported on the magnitude and nature of the resulting attrition. And, even 
fewer have dealt with random samples of the public which would allow other researchers to 
estimate what to expect in future general population surveys. An exception was Sobol’s (1959) 
reporting on the attrition that occurred in a five-wave panel studying economic attitude change. 
At Wave 1, in 1954, a probability sample of the non-institutionalized urban population of the 
United States was interviewed (n = 1,150). Subsequent waves were conducted six, 12, 18, and 
33 months later. Compared to the original sample, attrition for each subsequent wave was 17%, 
26%, 29% and 39%, respectively. 

Sobol reported that, in general, ‘‘because of canceling variations, the demographic struc- 
ture ... after five rounds of interviewing, remained very similar to that of the original [sample]”’ 
(p. 52). Yet there were some significant variations, with a disproportionate number of renters, 
lower income households, residents of large metropolitan areas, younger (under 25 years) and 
older (over 64 years) adults, and those not interested in the survey subject matter lost to the panel. 
Winer (1983) reported results of unpublished studies which generally confirmed Sobol’s findings. 
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It is important to note that in each of these studies interviewers knew Wave | respondents’ 
full names. In fact, in their article on techniques to minimize panel attrition, McAllister ef a/. 
(1973) stressed the importance of gathering detailed information about the respondent’s future 
whereabouts at the end of the interview, including ‘‘complete names and addresses of friends 
and/or relatives ... of the respondent’’ (p. 416). 

Although it can be argued that panel attrition is a serious enough problem to prompt re- 
searchers to obtain the full name and other identifying information of each Wave | respondent, 
this approach may cause problems of its own. In those instances where a respondent’s name 
is elicited as part of the Wave 1 interview, an explanation is sometimes given that the name 
is important because the respondent may/will be called back after some specified time to deter- 
mine if any changes occurred. This raises concerns about ‘‘evaluation apprehension’’ (i.e. reac- 
tivity) on the part of respondents (cf. Crano and Brewer 1973). Whereas some authors explicitly 
address the trade-off between attrition and reactivity (e.g. Sobol 1959), it is implicit in most 
other articles, that authors typically regard reactivity as less a problem than attrition. 

All of the aforementioned research was conducted with personal interviews. But what of 
panel attrition when telephone surveying is done, including those studies in which Wave 1 
respondents’ names are not recorded? In particular, what can be expected by a researcher who 
plans a priori to conduct a panel telephone survey and thus ask respondents for name iden- 
tifiers vs. a researcher who does not gather respondent name identifiers, either because he/she 
explicitly chooses not to or because a decision is made post hoc to convert a cross-sectional 
telephone survey to a panel after Wave 1 interviewing is complete? 

In an attempt provide a preliminary perspective on these issues, the present paper reports 
findings on the magnitude and the nature of attrition in two RDD (two-wave) panel studies 
conducted in the City of Chicago, each with a between-wave lag of approximately one year. 
It should be noted that these two surveys were conducted independently of each other, not as 
part a planned test of RDD attrition. As such, there are various differences in the substantive 
focus and specific execution of the two surveys, beyond the fact that in Study 2 a name iden- 
tifier was known for most respondents whereas in Study 1 it was not. We explicitly acknowledge 
that these differences in focus and execution somewhat limit the conclusions that can be drawn 
from the comparison of the two studies. 

For both surveys, one stage (i.e. simple) random-digit dialing was used to sample Wave 1 
respondents. In Study 1, respondent names were not asked as part of the Wave 1 interview and 
respondents were not told that they would be re-contacted. In Study 2, name identifiers were 
gathered at the completion of the Wave 1 interviews and were used to reach respondents at 
Wave 2. Respondents most often did not provide their full names, instead giving their first 
name only or other name identifier, e.g. nickname or initials. (Interviewers did not probe for full 
names so as to not contribute to possible feelings of paranoia on the part of reluctant respondents.) 

When respondents’ names are not known, how does one go about re-contacting the original 
respondent? This was a problem faced in 1979 by the first author when trying to determine 
the efficacy of creating a panel from a 1977 cross-sectional survey. As nothing was found in the 
published literature to provide guidance, a pilot-test was conducted with a resulting 50 percent 
of the 1977 respondents re-interviewed by asking for them by gender and age. 

The results of this pilot-test were encouraging enough to recommend the procedure for use 
in the first study reported here. In Study 1, interviewers dialed the same telephone numbers 
as the Wave | completions, verified each number whenever the call was answered, and informed 
the listener that approximately one year ago a person at the telephone number had completed 
an interview. The original respondent was identified by gender (e.g. ‘‘a man’’ or ‘‘a woman’’) 
and by age (e.g. “‘in his early twenties’’ or ‘‘in her late sixties’’). 
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In Study 2, a name identifier was known for more than eight out of 10 of the Wave 1 
respondents. For these respondents, Wave 2 interviewers asked for the respondent using the 
name identifier, after first verifying the telephone number. For the respondents with no 
name identifier, interviewers asked for the respondent by using demographic identifiers, as 
in Study 1. 

In reporting the results from these studies, it is our modest intention to shed preliminary 
light on the magnitude and nature of attrition in RDD panels. Although the results should not 
be generalized to a national RDD sample, they are suggestive. Given the prevalence of RDD 
sampling, we believe it is important to build a knowledge-base about the attrition that can be 
expected in panel studies where Wave 1 sampling is done via random-digit dialing, especially 
when researchers have no Wave | name identifier for respondents. By doing this, we can better 
consider strategies to reduce the size and effects of this attrition. 


2. STUDY 1 


2.1 Methodology 


In February, 1983, a city-wide (one-stage) RDD survey was conducted by the Northwestern 
University Survey Laboratory to gather baseline data for professors who were evaluating a 
series of community crime prevention programs in Chicago neighborhoods. (The question- 
naire took an average of 20 minutes to administer.) Approximately 2,800 telephone numbers 
were dialed in the process of completing 814 interviews. For each residence contacted, one head- 
of-household (male or female) was systematically selected as the designated respondent 
(cf. Lavrakas 1987; pp. 99-100). Whenever necessary, Spanish-language questionnaires were 
administered by bilingual interviewers. Up to seven call-backs were made to hard-to-reach 
respondents. Of all telephone numbers dialed, 1,247 were found to ring in eligible households 
(defined by the survey sponsors as English-speaking or Spanish-speaking households with at 
least one adult 19 years of age or older); those eligibles not interviewed either were unavailable 
at the time calls were made or refused to participate. 

One year later, in February, 1984, the Wave 1 telephone numbers were re-dialed to gather 
‘*nost-test’’ data for the evaluation project. In those instances where the telephone was answered 
within eight call-attempts (across different days and times), the following introduction was 
read by interviewers: 


Hello, isthis__- ss CC CUP My names. CCC and I’ m calling from 
Northwestern University. About a year ago (February 1983) we conducted an interview 
with a at this number. May I please speak with (her/him)? 


The interviewer first verified the telephone number and then gave her/his own name. The 
third blank contained pre-recorded Wave 1 demographic information (gender and age) about 
each respondent: e.g. ‘‘woman in her mid 30s,’’ or ‘‘man in his early 70s.’’ For those few 
respondents who had not given their year of birth at Wave 1, the third blank simply contained 
the gender identifier, ‘‘woman’’ or ‘‘man.”’ 


Once the interviewer was speaking to the original respondent he/she continued with the 
following explanation, before beginning the interview: 


The information you gave us last year was a big help in understanding the concerns of 
residents like yourself. We are calling back now to find out some things about the quality 
of life in Chicago neighborhoods during the past year. 
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The purpose of this statement was to reinforce the respondent’s willingness to cooperate 
with the Wave 2 interviewer by reminding the respondent of his/her cooperation in the Wave 1 
survey. 

Coinciding with the purpose of this evaluation project, respondents who had moved or 
changed their telephone numbers were not interviewed at Wave 2. This was due to the need 
to interview only those persons who resided at the same address as the previous year, since 
many of questions dealt with perceived neighborhood change since February, 1983. 


2.2 Results 


Due to a clerical error in processing the Wave 1 questionnaires and call-records, duplicate 
or incorrect respondent I.D. numbers were assigned to 17 Wave 1 respondents by the survey 
sponsors’ staff. For the purposes of this paper, these respondents were dropped from our 
analyses because we could not match correctly their Wave 2 dispositions with their respective 
Wave | data. Thus the following analyses are based on the 797 respondents whose Wave 1/ 
Wave 2 match was certain. 

The magnitude of the attrition. As shown in Table 1, approximately one-half of the Wave 1 
sample was re-interviewed (53%). Of the 375 respondents who were ‘‘lost’’ to the panel, the 
greatest proportion was due to telephone numbers that rang in a new household or in an original 
household from which the respondent had moved; this accounted for approximately 40% of 
the attrition. Second most frequent were those persons whose Wave | telephone number was 
no longer in service; this accounted for a fourth of those lost. Next in frequency of those lost 
were respondents who refused in some way. The fourth most prevalent reason for losing 
respondents were those who were never home during the Wave 2 field period when their 
telephone was answered, even after eight call-backs; (these 33 persons were verified to be the 
original respondent by someone else in their household). 


Table 1 
Dispostion of Wave 1 Samples for Study 1 (Names not known) and Study 2 (Names known) 


Study 1 - No names Study 2 - Names 
Wave 2 disposition Absolute Relative Absolute Relative 
frequency frequency frequency frequency 

% % 
No Wave 2 contact made 
Non-working, disconnected 95 11.9 94 9.4 
Never answered Ws ps 45 4.5 
Contact made 
Completion 422 52.9 $72 57.4 
Respondent gone from number 165 20.7 163 16.4 
Respondent never available 33 4.1 30 3.0 
Respondent refusal/partial 33 4.7 57 Sa 
‘*Gatekeeper’’ refusal 21 2.6 13 13 
Incapacitated, deceased 3 0.4 13 LS 
Misc. other 4 0.6 10 1.0 


Total 7197 100.0 997 100.0 
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The nature of the attrition. As shown in Table 2, the group of Wave | respondents in Study 1 
who were re-interviewed differed significantly on several factors from those who were lost to 
the panel. In terms of age, those adults less than 30 years of age at Wave 1 were re-interviewed 
with only 42% success vs. adults in the 40-59 year group of whom 60% completed Wave 2 
surveys. Blacks were significantly less likely to be re-interviewed than Whites. In terms of 
household income, those respondents who reported Wave | annual household incomes of less 
than $10,000 were re-interviewed with only 44% success vs. those with incomes over $20,000 
of whom 63% were re-interviewed. Married respondents were more successfully re-interviewed 
(57%) than those not married (49%). Sixty-two percent of home owners were re-interviewed 
compared with 47% of renters. The longer one had lived in the neighborhood and the more 
likely one reported at Wave 1 that he/she would not move, the more likely he/she was re- 
interviewed. 


3. STUDY 2 


3.1 Methodology 


During November and December of 1983, a city-wide (one-stage) RDD survey was conducted 
by the Northwestern University Survey Laboratory for professors who were examining eco- 
nomic well-being/hardship among Chicago families. (The questionnaire took an average of 
20 minutes to administer.) Approximately 3,900 telephone numbers were dialed in the process 
of completing 997 interviews. For each residence contacted, one head-of-household (male or 
female) was systematically selected as the designated respondent. Up to 20 call-backs were made 
to increase the likelihood of completing interviews with hard-to-reach respondents. In total, 
1,659 eligible households were reached; those eligibles not interviewed either were unavailable 
at the time calls were made or refused to participate. 

Sixteen months later (Spring 1985), all 997 telephone numbers were re-dialed to gather 
Wave 2 data. Unlike Study 1, in which respondents were not tracked if they had moved or 
changed their telephone numbers, an effort was made to find respondents whenever possible, 
although this effort resulted in only few successes as the respondent’s full name (first and last) 
was typically not available. As in Wave 1 of Study 2, at least 20 call-backs were used with the 
hardest-to-reach respondents. 

More than 80% of respondents had given a name identifier at Wave 1. This information 
was used by interviewers as follows: 


Hello, isthis Ss ? My nameiis____——S—_:— j and I’m calling from Northwestern 
University. About 16 months ago, in late 1983, we conducted an interview with a 
(man/woman) named at this number. May I please 


speak with (her/him)? 


As with Study 1, the interviewer first verified the telephone number and then gave her/his 
own name. The third blank contained the pre-recorded name identifier given by the respon- 
dent at Wave 1. Those respondents who did not give a name at Wave | were asked for by using 
the same procedure used in Study 1 (i.e. asking by gender and age, or by gender only). 


3.2 Results 


The magnitude of the attrition. As shown in Table 1, nearly six in 10 of the Wave 1 sample 
were re-interviewed (57.4%). Overall, the pattern of Wave 2 dispositions in Study 2 was very 
similar to what was observed in Wave 2 of Study 1. Of the 425 respondents who were “‘lost’’ 
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Table 2 


Respondent Characteristics of Wave 2 Re-interviews for Study 1 (Names not known) 
and Study 2 (Names known) 


Percentage Re-interviewed at Wave 2 
Respondent characteristic ee 
Study 1 Study 2 


Gender: 
Females 55 56 
Males 49 60 
Age: 
< 30 years 42* -- 
30-39 years 54 -- 
40-59 years 60 ~- 
>59 years 55 — 
< 34 years =o 54* 
35-49 years -- 64 
50-64 years -- 61 
> 64 years -- 50 
Race: 
Asian 68*** 42* 
Black 49 oe 
Hispanic 44 62 
White 58 61 
Education: 
Not high school graduate 45** 49** 
High school graduate a i 58 
Some college 50 53 
College graduate 60 64 
Graduate school -- 68 
Household income: 
< $10,000 44** oa 
$10,000-$19,999 55 -- 
$20,000-$29,999 63 -- 
$30,000 or more 63 -- 
< $12,000 on 50** 
$12,000-$17,999 a 59 
$18,000-$23 ,999 = 66 
$24,000 or more -- 64 
Marital status: 
Married = fe RY hes 
Divorced -- 68 
Separated ae 43 
Single oe 57 
Widowed -- mal 
Not married 49 oe 
Residential status: 
Own 627"* 59 
Rent 47 56 
Residential tenure in neighborhood: 
< 3 years 44*** -- 
3-9 years 55 -- 
10 or more years 56 - 
Likelihood of moving in next 2 years: 
Definitely will EN ig ng -- 
Probably will 47 =o 
Probably will not 57 =e 
Definitely will not 60 ~~ 
Note: Chi-square tests of significance were employed. 
*** ae 
+ < 01 
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to the panel, the greatest proportion (nearly four in 10) was associated with numbers that rang 
in a new household or in an original household from which the respondent had moved with 
no new number available. Second most frequent were those persons whose Wave 1 telephone 
number was no longer in service, accounting for nearly one in four of those lost to the panel. 
Next in frequency were respondents who refused in some way. The fourth most prevalent reason 
for losing Wave | respondents were those persons whose original telephone numbers were never 
answered at Wave 2. 

The nature of the attrition. As shown in Table 2, the group that was interviewed at Wave 2 
of Study 2 differed significantly on several factors from the group which was lost, with patterns 
similar to what was observed in Study 1. In terms of age, those adults less than 34 years of 
age and those more than 64 years of age at Wave | were least likely to be re-interviewed. Asians 
and Blacks were less likely to be re-interviewed than were Hispanics and Whites. In terms of 
education, those with less formal education were least likely to be re-interviewed. Those respondents 
who reported Wave | annual household incomes of less than $12,000 were re-interviewed with 
only 50% success vs. those with incomes over $24,000 of whom 64% were re-interviewed. Divorced 
respondents were most successfully re-interviewed (68%), whereas those who said they were 
separated at Wave | were least likely to be re-interviewed (43%). 


4. DISCUSSION 


4.1 Summary of Findings 


The two independent studies reported here were two-wave RDD telephone surveys, one with 
a 12 month lag between waves and the other with a 16 month lag. In Study 1, where names were 
not known for use at Wave 2, attrition was 47.1%. In Study 2, where name identifiers from 
Wave 1 were known for 83% of the respondents, attrition was somewhat less, at 42.6%. 

This marginal difference in attrition rates (x? (1) = 3.51, p < .10) is best considered 
within the following contextual differences between the studies: Study 1 respondents were not 
explicitly told at Wave 1 that they would be called back a year later and, thus, their names were 
not asked. In Study 2, respondents were told that they would be called back at some future time. 
Given the particular nature of the research in Study 1, no effort was made to track Wave 2 
respondents who had moved or changed their telephone number. On the other hand, an effort 
was made to do this in Study 2, although with little success. Study 1 employed a Spanish- 
language version of the questionnaire; in Study 2, Hispanics who could not speak English were 
not interviewed. 

In both studies, the vast majority of those lost to the panel were respondents who could 
not be reached via their Wave 1 telephone number, either because the number reached an 
entirely new residence, the respondent had moved from the household, or the number was no 
longer in service. 

Taken together, the findings of these two telephone studies are fairly consistent with past 
findings from in-person surveys (e.g. Sobol 1959) in identifying the types of persons most likely 
to be lost in panel studies. In both Study 1 and 2, younger and older adults, non-Whites, the 
less educated, and those with lower income were less likely to be re-interviewed than other 
demographic subgroups. 


4.2 Implications 


Given the cost/benefit attraction of RDD surveys, added to the analytic benfits associated 
with panel studies, it is worthwhile to consider options that may improve the representativeness 
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of the final panel in surveys that use RDD for Wave 1 sampling. But before discussing these 
considerations, the issue of asking for respondents’ names in telephone surveys merits further 
discussion. 

Asking for respondents’ names at Wave 1. As mentioned above, the issue is purportedly 
one of increasing the likelihood of reaching and, thus, re-interviewing the respondent at Wave 2 
vs. the possibility of creating an evaluation apprehension effect (Crano and Brewer 1973) which 
may bias Wave 2 data. Yet, more than this trade-off enters into consideration. 

The issues of confidentiality and informed consent also come into play: it is common prac- 
tice in academic survey research for a survey organization to never provide respondent telephone 
numbers to anyone, with the possible exception of the sponsor, and only when he/she is plan- 
ning a panel study or conducting follow-up interviews with respondents who have explicitly 
given permission for this. This practice follows from the reasoning that an assurance of con- 
fidentiality given to Wave 1 respondents is not violated when respondents are called back as 
part of the same on-going research. The fact that so few Wave 1 respondents refuse to par- 
ticipate at Wave 2, coupled with the observation that it is demographically predictable who 
is most likely to refuse at Wave 2, provides strong support for the conclusion that calling 
respondents back without having asked their permission at Wave 1 is not a problem. 

When a telephone survey sponsor can pay for the expense of tracking respondents who have 
moved, it appears logical to record respondents’ full names at Wave 1, since those who have 
moved may be tracked through telephone directories; calling new numbers given by telephone 
company recordings; or, even by calling former neighbors to get a forwarding telephone number 
in those cases where a respondent’s address is also known and a reverse-telephone directory 
is used. But if respondents will not be tracked at Wave 2, how useful is it to be able to ask for 
the respondent by name? 

It cannot be denied that interviewers say they prefer it. That is, most interviewers feel more 
comfortable asking for ‘‘John’’ or ‘‘John Smith’’ vs. asking for ‘‘a man in his mid-50s.”’ Yet 
the marginal difference in attrition rates in the two studies reported here, even considering the 
four-month longer lag time between waves in Study 2 which gathered name identifiers at 
Wave 1, does not provide compelling evidence of the advantage of names. We acknowledge 
that an unfortunate limitation of our paper is that other differences in these two RDD panel 
surveys may have contributed to the observed differential in attrition rates: e.g. Wave 2 call- 
backs were greater in Study 2 (eight in Wave 2 of Study 1 vs. 20 in Wave 2 of Study 2). Thus, 
this issue will remain unresolved until more controlled research is conducted. 

Given the current state of knowledge, we believe that it remains the responsibility of the 
individual researcher using an RDD panel to weigh the competing tensions of possibly biasing 
measures of the phenomenon under investigation by alerting respondents that they will be 
“‘measured”’ again (i.e. the ‘‘reactivity’’ effect) vs. the possibility of experiencing slightly less 
attrition by asking for names at the time of the Wave 1 interview. 

Considerations to minimize attrition effects. Some suggestions can be considered in the 
attempt to minimize the effects of RDD panel attrition. 

Sobol (1959) suggested the possibility of a Wave 1 over-sampling of those types of 
respondents who were most likely to be lost in subsequent waves. At first, this suggestion may 
sound appealing. This initial appeal follows the reasoning that if one knows who is most likely 
to be lost, then one can project an over-sampling of those groups at Wave 1. As was shown 
in Sobol’s work, and as found in the two studies reported here, one could estimate what types 
of persons should be over-sampled at Wave 1; e.g. older and younger adults. Over-sampling 
could be accomplished through the use of a screening procedure introduced late in the Wave 1 
field period; (although this clearly would increase Wave 1 total survey costs). 
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Although it is possible to over-sample, is it also desirable? In asking this question, one is 
ultimately asking whether the resulting panel is more than just an on-the-surface demographic 
match of the population of interest. In other words, is it enough to merely be concerned with 
getting, for example, the right number (i.e. proportion) of senior citizens in the final wave of 
a panel, or should one also be concerned whether one has the right ‘‘mix’’ of seniors? 

This is an empirical question that the present studies cannot answer. Clearly, more research 
is needed before survey researchers can be more certain whether it is preferable to over-sample 
at Wave | or to ‘‘compensate’”’ for attrition through statistical adjustments to subsequent waves 
of panel data. 

Another aspect of the attrition problem is associated with efforts to minimize the loss of 
those persons whom interviewers are able to re-contact at Wave 2; i.e. respondents who refuse 
or who are ‘‘never at home’’ to complete the Wave 2 interview. This type of loss accounted 
for 29% of the Study 1 attrition, and 34% in Study 2. What can be done so that interviewers 
might be more successful at minimizing these losses, other than merely employing traditional 
interviewer training techniques and making many call-back attempts? 

In this age of microcomputers it is quite feasible for interviewers to be given a Wave 1 
“‘profile’’ of each respondent, so as to be more familiar with the person to be re-interviewed. 
Care would have to be exercised to avoid creating expectations on the part of interviewers that 
might bias respondents’ Wave 2 answers. We are not suggesting that the interviewer necessarily 
use this information in verbatim form to identify the Wave 1 respondent; we believe name, 
gender and age are adequate for that purpose. But, there may be subtle changes in an inter- 
viewer’s verbal behavior that may lead to increased success at re-interviewing when the inter- 
viewer has a more detailed idea of ‘‘who’’ the respondent is. This suggestion must await testing 
before it can be confidently endorsed, but were it to prove effective without introducing bias 
into the data, it would be relatively easy to do. 

Similarly, introductory statements read by interviewers at Wave 2, could be targeted with 
special appeals to those demographic groups who appear most likely to refuse at Wave 2: in 
this case we are referring to the elderly, those with less formal education, those with relatively 
lower income, and especially those who were rated by Wave | interviewers as showing little 
interest and/or cooperation. Here again, a computer could be programmed to generate special 
Wave 2 introductory spiels based on Wave 1 data about particular respondents. 

These appeals must contain incentives for such persons to participate at Wave 2, as they 
are often persons with the least intrinsic motivation to participate in surveys. When planning 
for subsequent waves, surveyors should think of ‘‘why’’ such people would want to cooperate 
and work such reasoning into the interviewers’ introduction for these persons. Such introduc- 
tions may be lengthy and may even contain some rapport-building questioning. It may even 
be possible to give the prospective respondent some feedback about Wave 1 findings, without 
biasing Wave 2 responses. If so, the respondent may regard the re-contact attempt to be more 
of a ‘‘two-way’’ exchange. 

Regardless, computers could be used to generate these special introductions, which in turn 
would be matched only with those respondents for whom the message is targeted. Again, we 
have no empirical evidence to cite regarding the efficacy of this suggestion, but we believe it 
merits consideration and study. 


5. CONCLUSION 


Our findings suggest that attrition in RDD panels when respondent names are unknown 
is not of such magnitude as to render the surveying technique invalid or impractical. Due to 
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its nonreactivity, it would certainly appear to be the preferred approach in two-wave RDD 
panels in which the researcher has a priori reason not to want Wave | respondents to know 
they will be re-contacted. These findings also should provide encouragement for those who 
are thinking about converting an RDD cross-sectional survey into a panel. We hope that this 
primarily descriptive paper will encourage other survey methodologists to conduct and report 
the results of more controlled studies that investigate the nature and magnitude of RDD panel 
attrition, so that eventually, researchers can more confidently implement strategies to reduce 
the level of attrition. We suggest that this research should be guided by the observation that 
reductions in the magnitude of RDD panel attrition appear most likely to occur with well- 
organized surveying in which each respondent is approached as the individual that he/she is. 
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An Exact Test for the Presence of Stable Seasonality 
With Applications 


BRAJENDRA C. SUTRADHAR, ESTELA BEE DAGUM 
and BINYAM SOLOMON! 


ABSTRACT 


The X-11-ARIMA seasonal adjustment method and the Census X-11 variant use a standard ANOVA- 
F-test to assess the presence of stable seasonality. This F-test is applied to a series consisting of estimated 
seasonals plus irregulars (residuals) which may be (and often are) autocorrelated, thus violating the basic 
assumption of the F-test. This limitation has long been known by producers of seasonally adjusted data 
and the nominal value of the F statistic has been rarely used as a criterion for seasonal adjustment. Instead, 
producers of seasonally adjusted data have used rules of thumb, such as, F equal to or greater than 7. 
This paper introduces an exact test which takes into account autocorrelated residuals following an SMA 
process of the (0,qg) (0,Q), type. Comparisons of this modified F-test and the standard ANOVA test 
of X-11-ARIMA are made for a large number of Canadian socio-economic series. 


KEY WORDS: Standard Anova; Autocorrelated residuals; Seasonality. 


1. INTRODUCTION 


In the analysis of social and economic time series, it is traditional to decompose the observed 
series into four unobserved components, namely the trend, the cycle, the seasonal variations, 
and the irregulars. 

Socio-economic time series are often presented in seasonally adjusted form so that the 
underlying short-term trend can be more easily analysed and current socio-economic condi- 
tions can be assessed. There are several seasonal adjustment methods available which estimate 
the seasonal component present in a time series, but the Census X-11 variant (Shiskin, Young 
and Musgrave 1967) and the X-11-ARIMA method (Dagum 1980) are the most widely applied. 
To identify the presence of stable seasonality in a time series, the X-11-ARIMA method as well 
as the Census X-11 variant use the results of the usual F-test in a one-way ANOVA between 
monthly seasonal variations and the residuals. However, the residuals in this ANOVA are often 
autocorrelated, so the nominal significance level of the F-test may not be valid. Aware of this 
limitation, producers of seasonally adjusted data, do not guide themselves by the nominal 
significance level of the F-test for presence of stable seasonality but by some rule of thumb 
based on empirical knowledge (see e.g. Shiskin and Plewes 1978). In fact, implicit in the 
X-11-ARIMA test for the presence of ‘identifiable seasonality’ is that the F-value for stable 
seasonality should be greater or equal to 7 if moving seasonality is not present. 

The testing for stable seasonality (similarly for annual seasonal shifts) can be approached 
as a test for the significance of certain regression coefficients in a linear model with auto- 
correlated errors. The traditional Wald test, the likelihood ratio test, and the tests falling within 
a generalized least squares framework, all run into convergence problems in testing such a linear 


! Brajendra C. Sutradhar, Department of Mathematics and Statistics, Memorial University of Newfoundland, 
St. John’s, Newfoundland, AlC 5S7; Estela Bee Dagum, Time Series Research and Analysis Division, Statistics 
Canada, Ottawa, Ontario, KIA 0T6. Binyam Solomon, Directorate of Social and Economic Analysis, National 
Defence Headquarters, Ottawa, Ontario, K1A 0K2. 


154 Sutradhar, Dagum and Solomon: Exact Test for Stable Seasonality 


model with highly autocorrelated errors (cf. Sutradhar and Bartlett 1990). Pierce (1978) 
constructed an F-test based on transformed residuals which are approximately white noise. 
The transformation suggested in Pierce (1978) is equivalent to using the inversion of the error 
covariance matrix. But, the inverse of the error covariance matrix may not be obtained for 
highly autocorrelated errors. Recently Sutradhar, MacNeill and Dagum (1991) proposed a 
modified F-test, within a linear model framework for testing for the presence of stable 
seasonality. Their modified F-test is derived following Sutradhar, MacNeill and Sahrmann 
(1987), and the test accounts for the presence of autocorrelation in the residuals. The test does 
not require any transformation or any inversion of the error covariance matrix. 

Exact tests for testing the null hypothesis that the seasonal pattern changes over time against 
the alternative that the seasonal pattern is constant have been developed by Franzini and Harvey 
(1983). Unlike Franzini and Harvey, the present approach assumes that the seasonal pattern 
is stable over time possibly at different levels (due to annual shifts) and then tests for the presence 
of significant stable seasonality. 

In most empirical cases, a seasonal moving average (SMA) error model of the (0,g)(0,Q), 
type is sufficient. In this investigation we simplify the exact test proposed by Sutradhar, 
MacNeill and Dagum (1991), for such error models. The test is applied to examine for the presence 
of stable seasonality as well as of annual seasonal shifts in a number of socio-economic series. 

The plan of this paper is as follows. Section 2 presents the exact test. Section 3 analyses the 
results from the application of the modified F-test to a set of socio-economic time series 
and compares them with the values given by the X-11-ARIMA method. Section 4 gives the 
conclusions. 


2. MODIFIED F-TEST 


2.1 Selected Model 


Consider a stationary seasonal time series {Z,}, given by 
Z,= 8, + U,, (2.1) 


where Z, is the observed series at time f, S, is the seasonal component, and U, the irregulars. 
If the time series contains a trend, which is most likely, it is assumed that a suitable detrending 
technique will yield the model (2.1). In the latter case, the detrended series may be obtained 
from the original series by taking appropriate differences as in ARIMA modelling (Box and 
Jenkins 1970) or as is traditionally done by statistical agencies which use the X-11-ARIMA 
method or Census X-11 variant. 

Next, suppose there are k seasons in a year and there are kn observations in a time series 


of n years. Let Z{ (i — 1)n + j} bethejth (j = 1, ..., m) observation under the /th season 
(i = 1, ..., k) which corresponds to Z, in (2.1). We shall denote in similar manner the 
(7,/)th components of S, and U,, for allt = 1, ..., kn. Then, the model assumed for S, is 


(cf. Sutradhar and MacNeill 1989): 
SCG oh) nity o= perio hie (2.2) 


with Y iyo; = 0, Y 7, B = 0. 

The a’s and @’s in (2.2) represent, respectively, the stable seasonality and annual seasonal 
shifts in the seasonal time series. Thus, when testing for the presence of stable seasonality, we 
test the hypotheses 
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Ho: a = 0 vs. Ay: a; # 0 for at least one /; (2:3) 
and when testing for the presence of annual seasonal shifts, we test the hypotheses 
Ho: 8; = 0 vs. H,: B; # 0 for at least one /. (2.4) 


Consequently, the rejection of Hp in (2.3) and (2.4) would indicate that the series contains 
significant stable seasonality as well as annual seasonal shifts. 
Taking into account model (2.2), the model (2.1) can be written as 
LZ =) Xy t+ -U* , (2) 
where 


Z* = [Z(1), ..., Z(n), Z(n + 1), ..., Z(kn)]’, 
CeO) Ui) Oe el eo Uk) 


Y = [H,Q1, sey Ae —15 AK, B}, ae) Baraoe t- 
and X is the appropriate kn x (k + n + 1) design matrix. 


2.2 Test Statistics 


U* in (2.5) can be represented by seasonal autoregressive moving average (SARMA) statio- 
nary process (p,q) (P,Q),. In most empirical cases we found, however, that a (0,q) (0,Q), 
model is sufficient. Let }* denote the kn x kn covariance matrix of U*. Naturally, }* will 
contain 06 = (0;, ...,0,) and 0 =(0;, ..., 9g), where @ and O’s are the parameters 
associated with the SARMA (0,q) (0,Q), process. 

For the usual ANOVA model, viz., when the components of U* are i.i.d. N(0,o7), one 
tests the null hypotheses 6; = 0, and a; = 0 by using the classical F-statistics Fy; and Fy42 
respectively, given by 


Fa, = (k — 1)Q,/Q3, and Fy. = (n — 1)Q)/Q3, 


where 
n k 
Ones ui Zinse2 1 Onna atic Za? 
7=1 it 
and 
k n my: 
Osea hal ie mC ame acZ.):, 
i=l j=l 
with 
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Z;; being the jth observation under the ith season. In the present set-up, however, these 
statistics are inappropriate for testing the above hypotheses. This is because, the expected values 
of the sums of squares are affected by the dependence among observations. Also, sums of 
squares are not mutually independent. For the case when U* in (2.5) follow a SARMA 
(0,q) (0,Q), process, it can be shown that 


E(Q:) =k )) 8} + 07 (n — 1)C\ (8,8), 


E(Q>) 


jel 
k 
n ® a? + 0*(k — 1)C,(6,0), 
and iw 
E(Q3) = 07(k — 1)(n — 1)C3(6,), 
where, for example, for the SARMA (0,1) (0,1), process, 
C,(0,0) = (1 + 62)(1 + 0%) — (6,/6)(1 + 6%) (11 — 1/n) + (20,/n)(1 + 67) 
+ (6,0,/6){1 — 22/n — (n — 2)/n(n — 1)}, 
C,(0,8) = (1 + 67)(1 + OF) — 2(1 — 1/n)O,(1 + 67) 
+1/6{1 + (1 — 1/n)/11}0,(1 + 062) — (4/11)(1 — 1/n)6,6,, 
C;(6,0) = (1 + 6%)(1 + 62) + (20,/n) (1 + 67) + (6;/6)(1 + 62)(1 — 1/117) 
— (6,0;/6n) [n/11 — 2(n — 2)/11(n — 1) — 2). 


Consequently, the null hypotheses 6; = 0, and a; = 0 may be tested by using the modified 
F-statistics Fy, and Fy respectively, given by 


Fy = 4,(6,0)F 4, (2.6) 
Fyp = d,(0,0)Fa>, (2.7) 


(see also Sutradhar, MacNeill and Sahrmann 1987, Sutradhar, MacNeill and Dagum 1991), 
where d;(0,0) = C;(0,0)/C, (0,8), d,(0,0) = C;(0,8)/C,(6,0). The modified F-statistics 
Fy and Fy, account for autocorrelation of the residuals. 

Notice that in the independence case when 6 = 0,98 = 0,C;(-) = GQ, = (-) = C;(-) = 1, 
which is obvious. In that case the problem reduces to testing the hypotheses by using standard 
ANOVA F-statistics. 


2.3 Computation of p-value 


A simulation study (cf. Sutradhar and Bartlett 1989, Table IV, p. 1587) indicates that for 
the cases when k groups are independent, the distribution of the modified F-statistics for the 
SMA/(0,q) (0,Q), process, may be approximated by the usual F-distribution. In general, the 
F approximation to the modified F-statistic would be inappropriate, in particular when k groups 
are correlated and n is small. 
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In this paper we use the well known Satterthwaite (1946) approximation (cf. Sutradhar, 
MacNeill and Dagum 1991) to calculate the p-value, namely, P,(Fig, = Sin), where fy is the 


data based value of Fiz. In order to do it, we first compute the eigenvalues \,; => \, =>... = 
Ny Bea egg oe een Ap Ager = os pean OL 
Sih ds 9,8) Dinsesfrai Mca Day) Seni (2.8) 


where d(-) is given in equation (2.6), D, = R(RR’)'R’, with R = C(X’X)7!’, 
D, = X(X'X)~!X’, C being a suitable matrix obtained by expressing the: Hy : 6 }; = Oin the 
form Cy = 0, where y is given in model (2.5). In equation (2.8) J,,, is the kn x kn identity 
matrix. Then the Satterthwaite approximation yields 


PA(Fui = fui) = Pl Fa» = bd/ac), (2.9) 


where F,, denotes the usual F-ratio with degrees of freedom a and b, with 


o=(2 ») [E. = (¥ s) [Xe }. 


In equation (2.9), 


n 


r r n 
(at ya si Dg Eat | Deen 
J=S+1 


J=s+1 


Similarly, P,(Fiig = fy) may be calculated by using d,(-) and fp in place of d,(-) and fyy, 
respectively in equation (2.9). The construction of D, will now depend on a different C matrix 
which will be obtained by expressing the Hp : a; = 0 in the form Cy = 0. 


3. APPLICATIONS 


3.1 Monthly Series 


The modified F statistics Fy,; and Fin of equations (2.6) and (2.7) were calculated for a set 
of 26 monthly series obtained from various economic sectors, namely, Imports, Exports, Consumer 
Prices and Labour. All series cover the period January 1979 till December 1988 inclusive. 

Since the modified F-test is not valid when moving seasonality is present (except for annual 
seasonal shifts), none of the series selected are affected by moving seasonality according to 
certain preliminary tests available in X-11-ARIMA. (We also looked at the plots of the seasonal- 
irregular ratios.) 

The X-11-ARIMA method was applied to obtain the detrended series {Z,: t = 1, ..., 120}. 
Diagnostic checks show that the errors of the detrended series, U; (see equation 2.1) follow 
a (0,1) (0,1);, SMA model for each of the monthly series. The estimates 6, and 0, are used 
to compute the modified F-statistics Fyy, and Fy. 

In testing for the presence of annual seasonal shifts, the p-values for the modified F-test 
based on the Satterthwaite approximation and on the standard ANOVA F-test generally were 
found to be different. For both cases, however, the p-values were very large for each of the 
series indicating that there is no moving seasonality in the form of annual shifts. 
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Table 1 
Diagnostics of Stable Seasonality in Monthly Series 
Parameter 
; Estimates X-11-ARIMA Modified F Final 
Series oo F-Test? Fyp (p-value in %) Diagnostic® 
6; 0; 
IMPORTS 
1. Fodder and feed —0.09* —0.01 3.68 3.43(0.06) ¥ 
2. Coal related materials 0.02 -—0.01 64.40 58.76(0.00) ¥% 
3. Crude vegetable products 0.02  —0.07* 3.48 2.94(0.27) V2 
4. Wool & man made 
materials 0.02 0.29* 10.98 20.63(0.00) vi 
5. Precious metals UA bale UA L253 1.20(31.10) N 
6. Oils & fats O4ir* 0.01 8.59 8.22(0.00) ¥ 
7. Non-metal minerals 0.04 0.02 16.50 16.68(0.00) iy 
8. Aircraft engines 0:32 \, 0:00 Sar 2.36(1.79) N 
9. Other trans. equipments 0.19* —0.18* 3.48? 2.43(1.31) N 
EXPORTS 
10. Wheat 0.04 —0.03 1.89 P7181) N 
11. Asbestos 0.13* —0.03 6.83 6.15(0.00) be 
12. Wood pulp —0.27 0.20* 6.45 9.61(0.00) 4 
13. Textile fabrics 0:52") ) +O213* 12.05 15.06(0.00) ¥ 
14. Other fabrics 0.04 (Oy i 5.03 6.19(0.00) Y 
15. Television & 
telecommunication O12F%, ..0.01 9.26 8.99(0.00) Y 
16. Domestic export pass. —0.30* —0.14* 24.50 18.52(0.00) y. 
CPI 
17. Eggs —0.04 -0.01 6.90 6.50(0.00) ¥ 
18. Pasta —0.05* -—0.04 3.69 3.24(0.10) ¥ 
19. Onions —0.42* -—0.03 26.90 23.49(0.00) Me 
20. Housing 0.11* —0.34* 19.02 9.28(0.00) iyi 
21. Clothing 0.03 —0.42* 47.42 24.30(0.00) y: 
22. Transport —0.09* -—0.02 4.21 3.74(0.02) ¥ 
LABOUR 
23. Sask. employment 
(25-34) —0.19* —-0.11* 67.40 52.35(0.00) 2 
24. Sask. not in labour force 0.12* —0.36* 22.98 12.69(0.00) ¥ 
25. Ontario unemployment 
(25-44) 50:2 15) .0:07% 31.4 34.23(0.00) "4 
26. Ontario unemployment 
male & female (20-24) —0.02 0.19* 24.27 34.78(0.00) Y 


a Critical value is F(11,99; 0.01) = 2.47. 
b X-11-ARIMA and Modified F give conflicting inference. 
c Y (Yes) - stable seasonality is significant 
N (No) - stable seasonality is not present. 
* Significant values at 5% level. 


Survey Methodology, December 1991 159 


To test for the presence of stable seasonality, we computed the p-values of the modified 
F-statistic Fy (2.7) by using the Satterthwaite approximation and compared them to those 
given by the X-11-ARIMA F-test (which is equivalent to the standard ANOVA F,,,) for the 
26 monthly series. The results are shown in Table 1. 

The p-values of the modified F-statistic in Table 1 show that among the nine import series, 
three series do not have significant stable seasonality at the 1% significance level (the critical 
value of F (11,99; 0.01) = 2.47). Among the seven exports series, only one series, namely 
Wheat, appears to have no seasonality. All six CPI series have significant stable seasonality 
and similarly the four Labour series. 

The X-11-ARIMA F-test values give same results (either rejection or acceptance of the 
null hypothesis) as the modified F-test for a large number of series. It seems that for most 
of the monthly series, under the SMA (0,1) (0,1), error structure, the X-11-ARIMA F-test 
(or equivalently standard ANOVA F-test) is more affected by large negative values of 9), i.e. 
when there is seasonal autocorrelation in the residuals. This can be generalized by looking at 
the values of C;(0,0)/C,(0,0). By examining when this fraction is greater or less than 1, it 
may be seen that the direction of the inequality is affected by the signs of 6, and the size by 
the value of 6,. Only two series, namely, Imports Aircraft Engines and Imports other 
transportation Equipments, have standard F-test values which lead to contradictory conclu- 
sions with respect to the modified F-test. On the other hand, if we would follow the rule of 
thumb of F = 7 to justify seasonal adjustment, then the modified F-test would be in contradic- 
tion for eight out of twelve series. We then seasonally adjusted these eight series with the 
X-11-ARIMA method and found that the quality of the adjustment was acceptable for six out 
of the eight cases. All series passed the extrapolation ARIMA model automatically chosen for 
the program, six out of the eight series passed the X-11-ARIMA guidelines criteria for accep- 
tance; and the four series for which the Fy, values were relatively small, that is, falling 
between 3.24 and 3.74 were really strongly affected by trading-day variations. Only Imports 
Fodder and Feed and Imports Crude Vegetable products gave a seasonally adjusted output 
that could not be considered reliable. 


3.2 Quarterly Series 


The X-11-ARIMA method was applied to four quarterly series of the System of National 
Accounts to obtain the detrended values (z,;, ¢ = 1, ..., 40). It was found that for all four 
series U,; follow a (0,1) (0,1)4 model. The computation for the modified F-test is quite similar 
to the case for monthly series but since the covariance matrix )\* is different, the formulas 
for C;(-), Cy(-), and C;(-) in equations (2.6) and (2.7) were adjusted accordingly. 

Similar to the monthly series, the p-values for testing the presence of annual shifts 
based on the Fy, test were found very large and thus rejecting this pattern of moving 
seasonality. 

The results of the modified Fy, test and the X-11-ARIMA F-test for testing for the presence 
of stable seasonality in each of the four series, are given in Table 2. The p-value for two series 
namely, Deposits in other Institutions and Small Mortgages are not significant and in agree- 
ment with those obtained from X-11-ARIMA. Thus we conclude that these two series contain 
significant stable seasonality. For the remaining two quarterly series, the modified F-test and 
the X-11-ARIMA F-test give conflicting inferences. Contrary to the X-11-ARIMA F-test, the 
modified F-test yields significant p-values for these two series. Thus we conclude that these 
two quarterly series, namely, Net Financial Investments and Corporate claims should not be 
seasonally adjusted. 
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Table 2 
Diagnostics of Stable Seasonality in Quarterly Series 


Parameter ons 
: Estimates X-11-ARIMA Modified F Final 
Series ————— F-Test® Fyp (p-value in %) — Diagnostic® 
6, 0, 

1. Deposits in other 

institutions Oa" PA Bee 9.03 9.67(0.04) s% 
2. Net financial 

investments O87 SORT" 4.86" 2.56(8.16) N 
3. Small mortgages OTF —0.01 6.65 4.88(1.02) ¥ 
4. Corporate claims 0.77* —0.31* 7.88" 3.58(3.20) N 


a Critical value is F(3,27; 0.01) = 4.51. 
b X-11-ARIMA and Modified F give conflicting inference. 
c Y (Yes) - Stable seasonality is significant. 
N (No) - Stable seasonality is not present. 
* Significant values at 5% level. 


4. CONCLUSIONS 


This paper has introduced an exact test for the presence of stable seasonality and annual 
seasonal shifts based on the modified F-test by Sutradhar, MacNeill and Sahrmann (1987). 
The new test takes into account the possibility of autocorrelated residuals in the seasonal- 
irregular ratios of the X-11-ARIMA method. The residuals are assumed to follow a simple 
Seasonal Moving Average (SMA) model (0,q) (0,Q),. This test is applied to a set of quarterly 
and monthly series from the system of National Accounts, Imports, Exports, Consumer Prices 
and Labour. The residuals from the X-11-ARIMA method are found to follow seasonal moving 
average models (SMA) where either 6 and/or 0 were significant. The exact F-test gives values 
very different from those of the F-test in X-11-ARIMA (also in the Census X-11 variant) when 
the autocorrelation of the residuals is of a seasonal character, i.e., whenever 9 is significantly 
different from zero. 

Among the 26 monthly series analysed, only in two cases, the standard F-test values gave 
conflicting conclusions with respect to the modified F-test. On the other hand, if we would 
follow the common rule of thumb of F = 7 to justify seasonal adjustment, then the modified 
F-test gave contradictory results for eight out of twelve series. 

By looking at the seasonal adjustment output of these eight series we found that six can be 
soundly seasonally adjusted by the X-11-ARIMA method. 

Concerning the quarterly series, the modified F-test indicates that there is no stable season- 
ality in two out of the four series analysed. Furthermore, in one case, the F-test of X-11-ARIMA 
gives an F value greater than 7 whereas the modified F accepts the null hypothesis. 

It has been assumed throughout the paper that moving seasonality may be present in the series 
only in the form of annual shifts. The present test is not suitable to detect other types of moving 
seasonal patterns in the series. This raises the necessity of further investigations in this direction. 
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A Theory of Quota Surveys 


JEAN-CLAUDE DEVILLE! 


ABSTRACT 


Simple or marginal quota surveys are analyzed using two methods: (1) behaviour modelling (super- 
population model) and prediction estimation, and (2) sample modelling (simple restricted random 
sampling) and estimation derived from the sample distribution. In both cases the limitations of the theory 
used to establish the variance formulas and estimates when measuring totals are described. An extension 
of the quota method (non-proportional quotas) is also briefly described and analyzed. In some cases, 
this may provide a very significant improvement in survey precision. The advantages of the quota method 
are compared with those of random sampling. The latter remains indispensable in the case of large scale 
surveys within the framework of Official Statistics. 


KEY WORDS: Quota surveys; Super-population models; Restricted sampling; Regression estimation. 


1. INTRODUCTION 


Quota sampling is the method most frequently used in France by private polling institutions. 
It is easy to implement, inexpensive, and has many practical advantages. However, its disad- 
vantages are also well known: likelihood of bias, no possibility of processing non-responses, 
and the need for external information in order to set the quotas. In the English literature 
(Cochran 1977; or Madow et al. 1983, for example) quotas have a very bad reputation due 
to the lack of a reliable theory on which statistical inference can be based. The only ‘‘defenders’’ 
of the method (Smith 1983, in particular) base their arguments on the principles of inference 
conditional upon sampling, where the sampling plan may generally be ignored. 

This paper proposes a theory of quota surveys based on two types of modelling: population 
behaviour modelling (which is the approach of Smith or the ideas expressed in Gourieroux 1981), 
and modelling the method of sample collection, which may correspond to a more realistic idea. 

In both cases, variance estimates are obtained by resorting to variations of regression 
estimators. 

The first section of the paper describes the quota method and the results of the survey theory 
that can be subsequently useful. Parts 2 and 3 develop models for the behaviour of individuals 
in a population, or of those conducting the survey, which justify the method. The last section 
examines the problems raised, and attempts to demonstrate how the quota method can be used 
to add to the traditional probabilistic methods, rather than compete with them. 


2. A BRIEF REVIEW OF THE QUOTA METHOD AND SURVEY THEORY 


2.1 Cell Quotas; Quotas on the Margins of a Contingency Table - 
Some Practical Aspects of the Method 


At the simplest level, the quota method resembles stratified sampling. The distribution in 
the population of a discrete characteristic h possessed by N, individuals (h = 1 to H) is known. 


! Jean-Claude Deville, Institut National de la Statistique et des Etudes Economiques, 18, Boulevard Adolphe Pinard, 
75675, Paris Cedex 14, France. 
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The sample includes n,, individuals in category h; however, the choice of these individuals is 
left up to the those conducting the survey. The sampling rate f, = n;,/N, may vary from 
category to category. 

In practice, we prefer to control several criteria expressed as i, Jj, ...,h (i = 1 to J, 
j =1toJ,...,h = 1to#). Ideally, knowing the Nj, values of the multiple-entry con- 
tingency table allows the use of the previous method to define the number n,;__, of members 
in the sample depending upon the f;;__,, rates. Except in very specific cases (few criteria having 
few modalities each) this method is unrealistic, because it leads to a search for individuals who 
are extremely difficult to find. 

Thus, it is preferable to use marginal quotas, by calibrating the sample so that its distribution 
in accordance with the first criterion leads to a givenn;, , number of members, and the same 
is done for the other criteria. The only constraint on these marginal values is that they must 
be added to the overall sample size n. However, in practice, a single sampling rate f is adopted 
for each set of quotas: nj,...4 = SNi+...45M4j..4 = IN4;..4 andy. n = fN44.., with 
the obvious notations (+ in place of an index indicates the addition of all the modalities in 
the category represented by the index). 

Beyond the obvious collection advantages, this technique is the one most often imposed by 
the external data on which the quotas are based. These are obtained, for example, from various 
sources, thus preventing any cross-correlations. Another situation arises when the quotas are 
established on the basis of a large survey (a labour survey, for example): each distribution is 
done in accordance with a criterion (age, socio-professional category, etc.) that may be considered 
to be reliable. On the other hand, the cross-correlations are affected by a large random error, 
and cannot be used to set the quotas. 

In practice, the quota method is most often used to complement more traditional methods 
as the last sampling technique used in a multi-stage stratified survey on a geographic basis 
(region, size of the agglomerations). Each primary unit is assigned to a survey officer for whom 
quotas have been set. The survey officer also receives instructions to distribute his sample in 
order to make data collection as close to random as possible. 


2.2 Traditional Survey Theory 


We want to measure the total Y of a variable whose value \ for individual k is fixed, with 
no randomness. Only sample s is random, and the law of probability that governs s is known, 
since it is controlled by the statistician. Thus, we also know the possibility 1, that each indi- 
vidual will appear in s. Without any other information, the natural (unbiased) estimator to 
be used is the estimator based on inflated values: 


¥ = ND Y;,./ 1, = > dV, with dy = 1/7. 
kés Ss 
When the z;, are all equal to n/N, the sampling rate, we have: 
Y=N/n )) \& = NJ, 
AY 


where jy is the mean of Y in the sample. 


This estimator has a known variance, which is a quadratic form V( Y,) on the vector of 
Y, in the population: 
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Var(Y) = V(Yy) = yi) Y,(d, — 1) + ») Ye Yd) (aK, — 1} 7), (2.2.1) 
k kl 


where 7, is the probability of simultaneously having k and / in s. 
Similarly, the variance of Y can be estimated by a quadratic form on vector yYeot they; 
in the sample: 


VX) = ae Ani YY, 
kleés 


with Ap = (1.— ap)/ae fo k=l 


(ayy — 4, %))/ (aaa) ok I. 


Depending upon the sampling plans, these expressions take the specific forms found in the 
manuals (Desabie 1965; Cochran 1977; Wolter 1985). 

Any external information can improve the quality of the estimate. This is usually presented 
in the form of a vector X in which each of the p components is the total of a measurable variable 
in each of the possible samples. The estimate of Y can thus be improved by using regression 
estimation: 


Yreg = Y + (X — X)’B, 
where B is the vector of the coefficients of the regression of the Y, on the X;, estimated by: 


BS » (A,X, Xj)! ye, A,X ,Y,. 


Ss Ss 


When the constant is part of the regressors, or if it is a linear combination of the regressors 
and the sample has equal probabilities, the formula is simplified as follows: 


Yreg = X’B. 


The variance of Yreg is simply expressed by introducing the residuals of the regression 
E, = Y, — X;B into the population. We know that we have: 


Var( Yreg) = V(Ey) 


thus, we introduce in formula (2.2.1) vector Ey of residuals E,. At the same time, we approx- 
imate an estimate of this variance by V(e,), where e, is the vector of e, = Y;, — X{B, the 
estimated residuals of the regression. 

Under some sampling plans, these expressions assume particular forms. As a general rule, 
V and V are the positive quadratic forms, and the E;, or e, quantities smaller than the Y;,; the 
regression estimator leads to substantial improvements over the inflated values. 

A particularly important case that we will use later is one where_X is a vector of the total 
accounting variables (values on the basis of which the quotas are constructed). Typically, the 
additional information is the vector of dimension + (J — 1) + ... + (H — 1) formed 
by the quantities: Nj,4...4,N4j4...4,N4...4, fori = lto/,j = ltoJ — l,andh = 1 
to H — 1 (keeping only those variables that are linearly independent). Thus, the regressors 
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are the indicative variables of categories i(i = 1to/), jY¥ = l1toJ — 1),andh = 1 to 
(H — 1).Since the constant is a linear combination of the regressors (it is the sum of the first 
I of them), the regression estimator takes the form: 


Pres = Nigpi oe Arto N xtra Bashecte they Naan ini ogee 222) 
i j h 


where A; (for example) indicates belonging to category i. 
If we are only working with a single category, the regressors are orthogonal 2 by 2 and we have: 


Yreg = yD N;Y; 
i 


where Y; is the estimator of the mean of Y in category i. Thus, i. Yreg is nothing but the 
post-stratified estimator. 


2.3 Sampling Theories Based on Models 


In this approach, we consider that the Y, are random variables governed by a super- 
population model. This consists of parameters that we estimate on the basis of the sample. 
Wecan then calculate the probability, under the estimated model, of the non-observed values 
of Y, that is, Y,. The prediction estimator is the sum of the observed and predicted values and 
can be obtained as follows: 


Ypred = ye tet 1D fe 
Ss U-s 


If, for example, in an equal probabilities survey, the model is a regression Y,; = 
X;x + B + €,, €, when the k values are independent, centred, and of equal variance, and when 
the constant appears on the regression (or when we have a linear combination of X;, that is 
constant), we have: Y, Y¥, = },X;, 8; and the prediction estimator and the regression 
estimator are the same. 

We say that Y is without bias under the model when, for alls, &6(Y — Y) = 0(conditionally 
upon the sample, the probability and variance under the model are expressed as & and ‘V). For 
the prediction estimator, we must only have, for all k, the natural condition 6Y, = &Y;, in 
order for this to be true. With the model, we can also evaluate the average quadratic deviation: 
&(Yprea — Y)?, since we know that the two terms Yp,eq and Y are random, and that Yopreq 
depends upon sample s. The above-mentioned probability is thus conditional upon sample s. 
This follows a certain probability law already discussed in the previous paragraph. The preci- 
sion of this estimator can be measured by calculating: 


V.(Yprea) = E8(Yprea — Y)?. 


If the law of s is such that the Y; are independent (the so-called non-informative sampling), 
then this quantity equals: 


&(E(Yprea — Y)*), 


where the internal probability is conditional upon Y,. If Ypreq is equal to Ypeg, and we have 
a condition of independence, we will have: 


V(Yprea) = &(Var( Yreg))- 
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2.4 Comments on the Two Approaches Applied to the Quota Method 


a) In both cases, the process of estimation will be effective if the variable of interest is well 
explained by category indicators on which the quotas are roughly based, because the regres- 
sion adjustment residuals will be small. 

b) In a quota survey the ‘‘sampling plan’’ is not known by the statistician. Thus, he cannot 
make inferences without using a model. The latter may be a population behaviour model 
(‘‘model’’ approach) that requires him to assume certain responsibilities regarding the nature 
of what he observes. This approach will be developed in the second part of this paper. This 
may also consist of modelling the sampling plan; which means taking responsibility for the 
operation of the collection process. This approach will be developed in the third section of this 
paper. 

In all cases, the modelling speculation must be mobilized in order to validate a kind of 
inference. The question is to know whether it is easier and more plausible to model the behaviour 
of the individuals surveyed, or to model the sample collection process (including the contacts 
between interviewer and interviewee). 

c) In this respect, the hypothesis made in section 2.3 regarding the independence between 
randomness in the population and randomness in the collection process is crucial. If sampling 
is controlled by the statisticians, this guarantee can be ensured, except for the effect of non- 
responses. In the case of the quota method, there are no guarantees. Let us assume, for example, 
that we want to measure incomes Y;, the probability a, of finding k in the sample may be very 
low if Y, is large. In other words, the fact of belonging to the sample (which is 1 if k is in s, 
and 0 otherwise) and the residual of the super-population model €; are negatively correlated. 
This example illustrates well the main danger of the quota method, which the following theory 
does not take into account. 


3. QUOTA THEORY WITH A SUPER-POPULATION MODEL 


3.1 Cell Quotas 


There is a single cell category i = 1 to J for the known values N;. The model that can be 
imagined is as follows: 


Y; = Mm; + €, (3.1.1) 


e, centred independently of variance o? where i is the cell to which k belongs. 
The Gauss-Markov estimators of m; are the means observed in the various J; cells. Thus, 
the prediction estimator is: 


Ypred = 2 CN GON; by pects yy niyi = J Ni. (3.1.2) 
i i i 


I 


This has the form of the post-stratified estimator. Moreover: 


War(You —.)7 = iB o7N,(N; — 1;)/1;. (3.1.3) 
i 


This quantity does not depend upon sample s, as the latter always includes (with a probability 
of 1 !) n; individuals in cell i. 
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E&(Ypreq — Y)* can be estimated by replacing o?by its usual estimator s? = (n; — 1)7! 
Vices; (Ye — ¥j)* with s; being part of s in cell i. 

These results are from Gourieroux (1981) and represent, to a certain extent, a justification 
of the simple quota method. 


3.2 Marginal Quotas - ‘‘Representative’’ Case 


In this and the following paragraphs, we will restrict ourselves to the case of quotas overlap- 
ping 2 criteria iand /. The generalization with more than 2 criteria does not pose any particular 
problems, but leads to very complex notations that we prefer to avoid (see Appendix). 

Thus, the situation is as follows: the values N;, and N, ; of the two universe breakdowns 
are known. The sampling only allows samples of fixed size n = fN including n;, = fNj4 
individuals for each 7, and n,; = fN4; individuals for each /. 

We postulate an analysis of variance model in the population, formulated as follows: 

If k belongs to cell (i,/): 


1; =a; + B; Eh eet (3.2.1) 


The e, are centred, independent, and we have Var e, = 07 + 7}. 

For reasons of identification of the model, we postulate that 6, = 0. 

This is equivalent to postulating that Y, = (a; + uj) + (8 + vj) where uj, and vj, are 
independent, and their respective variances are 0? and T. 

We estimate a; and @; using the ordinary least squares (OLS) method, because we ignore 
the values of the variance elements; the @; and B; are solutions of the system: 


ny — ne ee = PO) 
/ (3.2.2) 
»s NijYij = n;B; ip iy nae (= tos), 

I . 


I 


with y;; the mean of the Y;, over the s;; part of the sample in cell (i,/) . Thus, the prediction 
estimator can be written as follows: 


Yoreda= be (Ni — ny) (&; + Bj) + Ne Mii. 
ij yj 
Result 1: Under model (3.2.1), the prediction estimator using the OLS is Ny. We check that 
it is unbiased for the model; that is, that 8(Ny — Y) = 0. 


Proof: Immediately from (3.2.2), and because of the fact that the quotas are proportional to 
the numbers in the population. 


Result 2: We have: 


&(Ny — Y)? = (N?/n)(1 — fon*( Poe AH #). 
i Pj 


This quantity does not depend upon the sample (as it depends only upon the quotas). Thus, 
to a certain extent, this is a justification for the marginal quotas method. 
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Proof: With m, = &Y,, using the unbiased character of the estimator we have: 


Pe 
&(Ny — Y)* = & (cin) Cle =a aah m)) 
S U 
2 
= & (cin) y Ee — ye «) 
Ss U 
= (Nin)* \).ny(o7 +77) —2(N/n) 0 my (op 77) 8 NG (oF Her7): 
ij yj Uj 
But 
SaNglo? chit a SoeN agian Nes & 
ij i J 
— (nin) ( a Nj407 + > n+; #) 
i j 
from which: 


&(N¥ — Y)? = (N?/n) (1 aa | yy) ny lor + 7) 


ij 


CN — 7) ( yy Piter +O) Ps; #) 
i 5 


The estimate of the precision of E (Ny — Y)’is derived from this. In fact, with this model, 
sj has a probability of o7 + 77. Thus, an unbiased estimator of the precision is obtained by 


Nin) audic 0) )s Buse 
ij 
if all the 1; are equal to or greater than 2. 
This estimator is formally identical to the one that we would use in a complete post- 


stratification on cells (i,). We can also use (N/n)? (1 — f) ¥, ef, where e; are the estimated 
residuals of the model. 


3.3 What Happens if the Model is False? 


3.3.1 Aninitial way of looking at the question is to put model (3.2.1) into the general model 
where the mean of Y, depends upon the pair (i,j). This can be written as follows: 


Ye = Ot By + yj + x; (3:34),)) 


with the usual hypotheses for e, and the terms of interaction y;; that verify the constraints of 
identifiability: 


yD Ni ¥ij = 0 and 3 Ni Vij = 0. (3.3.1.2) 


J I 
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Thus we have: 


S(Ny — Y) = Dy (Nnjj/n — Nj) vij> (3.3.1.3) 


y 


such that the estimator is biased for the model except when n;; = fNj; , which has no reason 
to exist. 

This means that the terms of sum (3.3.1.3) may well compensate for each other, since their 
signs are a priori undetermined. 

On the other hand, if ‘‘good’’ sampling precautions are taken, Nn;;/n — N,; should usually 
be close to 0. 

It is clear, in any case, that the more suitable the additive model is (small y;; ), and the more 
the sampling plan approaches randomness, the more likely it is that bias will be reduced. 


3.3.2 Another way to view the misrepresentation of the model, which has already been 
described, is to no longer admit that there is independence between the randomness of the 
sample and the randomness of the additive model. This means that distinct models should be 
developed for the (¥, k€és) and (Y;, /¢s) vectors. This approach has often been used in the 
econometric literature, to which the reader is referred. It is clear that risk-taking in regards 
to the data becomes enormous, and is often incompatible with objective work on the part of 
the statistician. 


3.4 Marginal Quotas with Unequal Rates 


In the case of cell quotas, we can arbitrarily set quotas for each cell. Until now, in the case 
of marginal quotas, we have only examined the case where the quotas were proportional to 
the size of the population. 

In many cases however, we may be tempted to over-represent certain categories. If, for 
example, we want to study household assets, we may want to set the largest quotas for older 
households (quotas by age group), on the one hand; and for those where the head is self- 
employed (quotas by social categories), on the other. 

Thus, we formally force the sample to fall within a given size n;, and n,; (however, the 
sum of n;, is always equal to the sum of n,;). 

In this case, always using the OLS as an estimation technique, we can easily find that the 
total prediction estimator is: 


Ypred ae Dy N;4@; + 3 Ni; Bi. (3.4.1) 


U ff 


a; and B }; always verify estimating equations (3.2.2). It is easy to see that this estimator may 
be expressed as follows: 


ee 1 2 pee ae 
Yprea = ye (wi? + wi”) ny¥y = se Ni Jij- 
= - 


ij 


Thus, the quantities (w{') + w{?)) nj seem to be estimates of the size of cells (i,j), an 
idea that will be largely exploited in the following sections. 

On the other hand, the variance of this estimator under the model depends upon all the nj,j, 
and this can be demonstrated by a rather cumbersome calculation. The justification of the quota 
method described above no longer works. 
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4. MODELS FOR THE SAMPLING PLAN 
4.1 A Model Sampling Plan 


The idea is one of a simple random sampling constrained by the quotas imposed. The selec- 
tion algorithm, while totally unrealistic, consists of drawing a series of simple random samples 
until we find one that verifies the quotas. Thus, each sample that verifies the quotas has the 
same positive probability of being drawn, the samples that do not verify the quotas have a zero 
probability of being drawn. 

The purpose is to model the fact that the person conducting the survey will correctly follow 
the dispersion constraints on the survey units assigned to him. 


4.2 Cell Quotas 


This sampling model is based on an a priori stratification. Its practical advantage is that 
it does not require a sampling frame where the stratification variables are present. It is 
implemented rigorously in certain cases, for example, in a telephone survey based on a non- 
informative random list of telephone numbers, and when surveys are carried out only until 
the quotas are met. 

The formulas that provide the estimators, the variances, and the precision estimates are those 
given in all the manuals. They have a certain similarity with those described in section 3.1 
(see Gouriboux 1981). 


4.3. The Case of Marginal Quotas: General Estimators 


The sampling model is that of simple random sampling constrained by marginal quotas. 
SRS provides samples with n,;; members in the various cells that can be taken as a random 
vector (in whole values) in R”. The quota constraint means that we are limited to a random 
vector as follows: 


yy ny = n+ (= 1 to 1) and ty ee — | to d=), 
j i 


that is, one that varies within a sub-space of size JJ — IJ — J + 1. Weplace ourselves in the 
case where the overall sampling rate is negligible, and the law of the n,; can be compared to 
a multinomial law (n, pj = Njj/N). 

Conditional upon nj, the ¥;; estimate the Y;; without bias. The idea is now to construct an 
estimator of the total of Y by weighting the y;; by the estimators of Nj, that is, the p,;. If we 
choose to maximize the probability, this is proportional to: 


[1] 2%. (4.3.1) 
i 


Thus, we maximize 


yy ny Logpy (4.3.2) 
i] 


under the following constraints 


Ne Py Pi RPS Mtowl)p and Win =p4j(J=1toJ—1) (4.3.3) 
yi i 


172 Deville: A Theory of Quota Surveys 


which leads to solving the system for a;, b; (pi+ = Ni+/N D+; = N4;/N are known): 


>) bi (a,+ b)-'=p, (i=1 tod) 
(4.3.4) 


yy By (a; + bj)! = Daj (j= | to. — bab; =. 0), 


with p;; = njj/n frequency in the sample. 


The estimators of p;; are thus fj; (a; + b;) ~! and the estimator we are looking for can be 
written as follows: 


Yo = (N/n) YY njj(a; + b))~* Iz = (N/n) yy weYes (4.3.5) 


ij Ss 


where we = (a; + Dj) ~1 is the weight added to Y, in the case when k appears in cell (i,/). 
This estimator is asymptotically without bias under the SRS model in U, as are the maximum 
probability estimators. The quotas do not play an explicit role in (3.3.4), but they affect the 
values of a; and bj. 

In the normal case when the marginal quotas are ‘‘proportional’’, with a fixed sampling 
fraction f, the solution of equations (4.3.4) is evident: a, = 1 for any i, and b; = 0 for any /. 
The estimator of the total is Ny, as could be expected, and has the same expression as the equal- 
probability probabilistic sampling. 


Comment: The use of maximum probability to estimate the proportions is rather arbitrary. 
A chi-square criterion (minimize ¥ ,; (pj — 6j;)*/pj;) would make the (4.3.4) system 
linear. 


4.4 Variance of the Estimator and its Estimate 


4.4.1 To establish a variance formula we will use the parametrization of variable Y used by 
J.C. Deville and C.E. Sarndal (1990), which we will express in the form of a: 


Lemma: For any variable Y = (Y,;k€U), we can choose an uniquely defined parametrization 


¥y, = Yj + Ry if k isincell (i,j) (kKeUy) with \ Ry = 0, 
ke Uj; 


Yj, = Aj + B, + Ey with B, = 0 


Leto 


II 


j 


ltoJ—TJ. 


i 


In fact, A; and B; are numbers that minimize the quantity Yy (¥, — A; — B;)* where, in 
an equivalent manner Yj; Nj (¥;; — A; — B,)?. 
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Thus, we can write: 
Yo = (N/n) ys Nnij (a; ate bj) ~' (A; ar Bj =F Ej; Se Ri) where Ri = 3 Ri/ Nig 
a Sij 


Taking into account equation 4.3.4 and the lemma: 


Yor. )hidy (Ey Bet with Ny = N/a) nla; top)" (4.4.1) 
ij 
which is the basic expression for the calculation of the variance. 


Conditional upon n,;, the Ni are constant, and sub-samples s;; are independent simple 
random samplings. Thus we have: 


Cond bias ( Yo) = 2 NE hv Af ie Pabs 
ij ij 
Cond Var(Yo) = )) NiVij/ny where Vij = (1/Nj) )) Ri. 
Uj Uj 
Thus (demonstration in the Appendix) we have: 
Result 1: 


var ( 2 bs Ei) = 1/n 3 pi Ei; - 
ij z 


ij 


Furthermore, the probability of 6j(a; + b;)~' is (in terms close to 1/n) pjj(a? + b?) 7! 
where a; and b? are the solutions to equations (4.3.4), in which fj; are replaced by the exact pj,j. 


This leads to: 


Result 2: The variance of the quota estimator Yo is given by: 


Var (Yor —PeN mye ye py eg (are) Va) 
ij 
If the quotas are proportional to the size of the population, we will have: 
Var(Yg) = (N*/n) ) pi(Ei + Vij)- 
ij 
4.4.2 Estimating the Variance 
The conditional variance of Yo can be estimated by: 


ys Ni; Sij/ ij — (N?/n) y Bij (Qj + bj) 7! si, 
yj ij 
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where si is the usual unbiased estimator of Vj;;. The probability of the square of the conditional 
bias is (N?/n) Yj, pj Ej, and is estimated by (N?/n) Yj 6 Ej; where E;; = 9, — A; — Bjand 
A; and B; are the solutions of: 


ae Bi (A; + B;) 


J 


2D Pg Sint B10 
i 


(4.4.2) 
\. 6 (A; + B) = Yo by, G=1toJ—D with By =0. 
J J 


In other words, the estimate of Ej; is obtained by fitting to the data an additive ANOVA 
model without interaction, the fitness criterion being that of least squares weighted by 
(a; 5 bj) 7 : 

Thus, the variance estimator is: 


Var( Yo) = (N?/n) Y) By (BG + (a; + bj) —1 5%). (4.4.3) 
ij 


When the quotas are proportional to the population numbers, this expression can be 
simplified as follows: 


(N?/n) \) ny (EG + si)/n. (4.4.4) 
ij 


If the n;; are all sufficiently large that n;/(mj — 1) = 1, the sum of the formula is the sum 
of the squares of the residuals estimated in the OLS adjustment of the XY, = A; + Bj + 
residual model. Thus, the estimation procedure is simple: 


© use the OLS to fit the additive model to the individual data 
® create the variable e, of the estimated residuals 
© Var(¥o) = (N?/n) + (1/n) Ye. 


This formula is precisely that proposed in paragraph 2, and based on the super-population 
model. A rather neat situation! 


4.4.3 Discussion of the Results 


The variance breaks down into two parts: one that can be seen as the probability of the square 
of the conditional bias; and one as the probability of the conditional variance. 

The first term does not depend upon the quotas imposed on the sample, but only upon the 
quality of the fit of an additive model to the variable of interest. This part of the variance is 
diminished by choosing quota criteria that can best explain what we want to measure. 

The second term, on the other hand, depends upon the remaining variability (Ni; Vgl ity) 
and the number of observations collected in each cell. Since the size of the sample is fixed, 
we must attempt to make the nj; as close as possible to Neyman’s distribution: nj; « Nj Vij ia 
This may be achieved approximately by overloading quotas n;, and n,;, which correspond 
to large values of V;;. Thus, in some cases, it is possible to improve the precision of a quota 
survey considerably. 
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4.5 Combination of the Quota Method and Stratified or Multi-Stage Samplings 
4.5.1 The Case of Stratified Sampling with a Quota in Each Stratum 


If the size of the criteria used to set the quotas are known in each stratum, the method 
described above makes it possible to construct an unbiased estimator, under the hypothesis 
that sampling functions like an SRS constraint in each stratum. If the allocation of quotas is 
proportional to the size of each stratum, the estimator is the natural estimator of the stratified 
sampling. If ‘‘national’’ quotas are used with each stratum, a correction should be made by 
reweighting. 

On the other hand, if the size of the quota variables is unknown at the stratum level, it is 
not possible to correct the estimators to eliminate ‘‘structure effects’’ related to the stratification. 
Since, furthermore, the purpose of stratification is to construct dissimilar sub-populations, 
the corrections required will generally be quite large. Thus, the quota method is not recom- 
mended (except when the validity of the additive model is quite clear, cf part 3). 


4.5.2 The Case of Two-Stage Sampling 


Let us assume a two-stage sampling (inside a stratum where the sizes of the quota variables 
are known). If the sizes of the quota variables are known at the level of each primary unit, 
there are no problems. The theory in section 4.4 makes it possible to obtain an estimator of 
the total Y in each primary unit, as well as to calculate its variance, and an estimator of the 
latter. These quantities can then be used to obtain an estimator of Y, as well as an estimator 
of precision (cf Rao 1975). If the sizes of the quota criteria are not known at the level of the 
primary units, but only at the stratum level, we again have a problem that is impossible to 
correct. However, there is generally little harm if the PU are relatively similar: the structure 
of each PU is close to that of the stratum as a whole, and the corrections to be made for each 
PU are close to those that must be made at the stratum level. 


4.5.3. In Conclusion 


In conclusion, in the case complex multi-stage stratified sampling, the quota method may 
be used as the final sampling method if the stratification was carried out effectively by 
regrouping the similar primary units together, and if quotas derived from the data relative to 
each stratum are used with each PU. 

To the extent that the hypothesis of simple random sampling constrained in each PU may 
appear to be quite satisfactory, the quota method is justified independently of any super- 
population model. 


5. CONCLUSIONS AND PROBLEMS 


5.1 How Should Non-response Be Taken into Account? 


As we have already shown, this is the most important limitation in our theory. As far as 
sampling using the quota method is concerned, we do not have, in principle, any information 
on members of the population who refuse to respond to the survey, and we find ourselves 
lacking individual information on the subject of non-respondents. However, the situation is 
not as desperate as one might think. Let us illustrate this using a very simplified example. 

We have carried out a simple quota survey using a sample of ni individuals in category / 
with a population N;. An acceptable model of non-response postulates a response probability 
of r. if an individual belongs to category c with a population N.. The (unknown) population 
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of the intersection between quota category j and class c of the non-response model is expressed 
as N*£. The population likely to respond in category jis thus N,; = ¥. N§r,. By setting a quota 
n;in this category, within the framework of model (4.1), we obtain a probability of inclusion 
in the sample of w; | = n,/N,;. In the sample, we collect n{ individuals belonging to the 
intersection (i,c) between the two categories. This quantity is random, and its probability is 
N‘r.w;_'. If we attempt to estimate NS, we will solve the estimating equations derived from 
the following relations: 


Chex é -1 
Ni =niWre 


c 
Dare | ise Bias 


Thus, ranking ratio technique makes it possible to obtain estimates of 7, and W;, and to 
derive estimators Nf = n{w;f.' from the sizes of the intersection (i,c). We can also obtain 
an estimator of the total of Y: 


% ee Cx Cee —1 csc 
Yxr = diy Ni = DD Tee Wi NPY; 
ic 


ic 


where ¥; is the mean of the Y; values in the sample in category (i,c) . Thus, estimation tech- 
niques based on fitting should allow for the honourable processing of non-responses in quota 
surveys. 


5.2 Some Points of Comparison with Probabilistic Surveys 


Regardless of how we try to understand it, the quota method demands the formulation of 
a hypothetical model to fit the data. On the other hand, a probabilistic survey does not, in 
principle, depend upon any model. In practice, sampling for a probabilistic survey is a model 
to which the reality of data collection attempts to conform. In fact, we are well aware that, 
in any probabilistic survey, some compromises of detail must be made with the model (necessary 
exclusion of certain units, replacement of others after selection but before data collection, efc). 
However, we can say that statistical biases are always much lower in probabilistic selection 
than when using the quota method. On the other hand, quotas make it possible to use, in the 
sampling stage, additional information that cannot be mobilized in a probabilistic selection 
process. As a result, the variance of a quota sampling is similar to that of a regression estimation, 
and is thus generally smaller than that resulting from a probabilistic survey associated with 
its estimate of standard inflated values. The choice is between bias due to the model associated 
with low variance, against lack of bias. Two types of conclusions can be drawn from this 
approach: 


5.2.1 Precision depends mostly upon the size of the sample. On the average, in the case of 
small samples, probabilistic sampling will produce the worst results; and the bias of a quota 
survey will be more tolerable than the lack of precision of a probabilistic survey. For large 
samples, on the other hand, the quota method will have a clear bias that is obviously incom- 
patible with the confidence interval without bias of a probabilistic survey. 
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Where should the boundary between the two methods be set? It is hard for the theory to 
be specific. On the other hand, experience in the French institutes may lead to a solution to 
this question: most national quota surveys are carried out on samples of 1,000 to 2,000 
individuals. On the other hand, no national probabilistic survey mobilizes less than 5,000 units. 
It would seem fair to say that a size of 2,500 to 3,000 surveys is a practical boundary between 
the two types of surveys. 


5.2.2 Official Statistics or Marketing 


In a survey, the use of any speculative model represents methodological risk-taking. This 
may be perfectly reasonable if the users are aware of it, and if they have ratified the speculations 
leading to the specification of the model. This is typically what happens, at least implicitly, 
in marketing surveys: an organization, company, administration, or association requests a 
sampling survey from a polling company. A contract marks the agreement between the two 
parties respecting the implementation of the survey, its price, the result delivery schedule, and 
the methodology used. In this methodology, models are used to formalize the sampling or 
behaviour of the population. Thus, from this point of view, the use of the quota method may 
be quite proper. 

Official statisticians, on the other hand, are responsible for generating data that can be used 
by the entire society; and that can be used, in particular, in the arbitration of disputes between 
various groups, parties, and social classes. The use of statistical models, particularly 
econometric models that describe the behaviour of economic agents, may turn out to be very 
dangerous, partial, or affected by a questionable or disputed economic theory. Official statistics 
should not tolerate any uncontrollable bias in its products. It should carry out sample surveys 
using probabilistic methods. 

There is no real opposition between quota survey techniques and those using controlled 
randomness, quite the opposite - they are complementary. As a proof of this, the statistics 
that are used to construct the quotas are themselves very often derived from large surveys carried 
out by the National Statistics Services. However, quota survey technicians find it hard to admit 
that these data are obtained using methods other than traditional, confirmed, and well-founded 
probabilistic techniques. 
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APPENDIX 
Demonstration of the Results of Section 4.4 


1. Notation and Results 


In order to deal with the question in a general way, we will require certain convenient 
notations. We have Q qualitative variables whose modalities are indicated by using indices from 
1 toZ,whenqg = 1toQ. A “‘cell’”’ is denoted as c; that is, a series of O indices where the qh 
could have a value of 1 to J,; and q, is the value of the q'" index (q" projection of c); in a finite 
population U, of size N, U.is the population of individuals in cell c, when the size of the cell 
is N.. The quantity N;*? = ¥, gq. = §N- is the total of the Q-dimensional contingency table 
where the cells are represented by c for the i‘ modality of the g" variable. If we postulate that 


s 1 
ear yy Y¢: 
© keUe 


We will obtain the following results: 
Result 1: Variable Y,(k€U)may be parametrized by the following numbers: A hi E..and R, by: 


Y,= Y.+ R, if keU.. Wehave > R, =.0; for. any .c. 


Ue 
Q 
a= Aj. +E, with Aj, = 0 [Orpen 100. and 
q=!l1 
yy NE =0 for g=itoQ and i=1 tok. 
Qc=i 


These numbers are obtained from the minimization of: 


Let us assume that we have a sample s. We will use 1 to denote all quantities in the sample 
that are similar to whatever we have already indicated in the population. 

We assume that s was obtained on the basis of simple random sampling (with or without 
replacement) in accordance with an equal probability scheme constrained by the totals n;*? 
(q = 1toQ,i = 1 to J,), the quotas. 

The purpose of this appendix is to demonstrate the following result: 

Result 2: The variance of 5. N, E, is approximately equal to 1/n ¥. N. E2 when n, and N/n 
become arbitrarily large. 

The following section will provide a more precise formulation for this result. 


2. Sampling Plan and Asymptotic Reduction 


Let us consider the following two sampling models SR and AR: 


SR: Bernouilli Sampling. Each of the units of N belong to s with a probability f, and the N 
drawings are independent. 
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AR: Each unit is drawn a number y, of times; y follows Poisson’s law with f parameters. The 
v, are independent variables. 
A simple random survey without replacement (SRSWOR) of fixed size n is an SR sampling 
if the total size of the sample is n. 
A simple random survey with replacement (SRSWR) of fixed size n is an AR sampling when 
we have 7 observations; that is, when );, y% = n. 
In the case of SR sampling, the law of the vector n, is obtained as follows: 


Pretns) = II (3%) rol Sf) “orellen 


c 


In the case of AR sampling, we have: 


INAS 
Pr({nj) = |] re exp(— fN-). 


c 


In both cases the variables n, are independent. 
In the case of SR sampling constrained by ¥n, = n, the law of the n, is hypergeometric: 


a 
Pr({ne}) = T] ) (*) 3 


In the case of the restricted AR sampling, the law is multinomial: 
Pr({m.}) = [J pees/ne!. 
¢€ 


The sampling plan model retained by the quota method described in paragraph 3 corresponds 
to constraints on these two schemes; which is equivalent to constraints on the SR and AR plans. 

If we assume that N tends toward infinity, that f tends towards 0, and that n* = fNtends 
toward infinity, then in the two plans, the law of theu, = n*~” (n, — fN.) = n*” (p* — Dp), 
with p* = n,/n*, tends toward a multidimensional normal law with independent u,, with zero 
probability and variances equal to p,. 


3. Proportional Sampling 


In this case, we have N. = N/n n-,, so that the quantity for which we want to determine 
the variance is: 


N 
> me uF, 
A Dy Heke 
n* 
c 


where the vector of the u, follows a centered normal law with a diagonal covariance matrix 
A = diag(p,), constrained by the relationships expressed by the quotas: 


3 be 00nd — 110.0, j= 4 fort. it gq = 1)7i-='1 ford, — 1 if g¢,=2-for Q. 


Qc=l 
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If we let U represent the vector of the u,, the relationships can be written as follows: 
AUr=n0; 


with A matrix with / = },/J, — (Q — 1) rows and k = II, /, columns, where | and 0 
represent the constraints. This also expresses the fact that U varies in the kernel L of the operator 
defined by matrix A. The (asymptotic) law of U is thus that of a centered gaussian vector W 
with a matrix whose covariances equal A, when AW = 0. Thus, it is a question of evaluating 
the variance of a scalar product U’E, where E is the vector of the E,. 


It is important to emphasize the following two points: 


e The constraints upon the £, given in result 1 can be expressed on the basis of matrix analysis 
by AAE = 0. In other words, AE is a vector of L = KerA, or a vector of Ker(AA). 


¢ Let P be the projection of ®* on L orthogonal in the A~! metrics. P verifies the following 
relations: 


ONCE, Pra eli ha. Lb 
o Py = 0 © wxeL, x’A7!y = 0; KerP = A(ZL*), 


where L* is the supplementary line orthogonal to L in the natural metrics. 


The gaussian vectors PW and (1 — P)W vary in L and A(L*) respectively; and their 
sum is equal to W. Moreover, they are independent; in fact, their covariance matrix is 
E(PW)((1 — P)W)’ = PA(1 — P’). Thus, P’ is the kernel projector L* and can be repre- 
sented as A(L+)+. The image of the projector (1 — P’) is thus Z+. That of A(1 — P’) is 
A(L*); that is, the kernel of P, g.e.d. 

At this point, we have to evaluate the variance of Yu, FE, = U’E. Thus, in accordance 
with the previous statements, we can write W = U + V, when Uand V are independent. The 
law of W conditional upon W€L is none other than the law of W conditional upon V = 0. 


Moreover, we have: 
WE =\(Ao 4) SOAP». 


Since AE is in L, and V varies in A(L*), the scalar product above is zero. From this, we 
can deduce that: 


Var(U’E) = Var(W’E) = E’AE = Y) p, Et. 


c 


The asymptotic variance of is thus equal to N/n* Yn, E. 
N? N 
— Pe B2 = — Yl NeEE. 
ce Cc 


4. Sampling using ‘‘Non-Proportional’’ Quotas 


Let us complete the preceding asymptotic reduction. Now, the vector f° of n,/n* is 
constrained by 


Ap® = Ap + n*-AA¥N,, 
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where Ap is the vector (1-dimensional) of the ‘‘proportional quotas’’, and Vp is the only vector 
(k-dimensional) of A(L*), so that A(p + n*~” Vo); that is, the vector of the quotas imposed. 


Thus, as in the previous paragraph, U. = n*” (fo — p) may be analyzed as a gaussian vector 
W = U + Vconditional upon V = VY. Thus, EU = Wo, and the covariances matrix of U 
is the same as that of U. 

Moreover, we go from f° to f by estimating the maximum resemblance. Under asymptotic 
gaussian conditions, this consists of minimizing the quadratic form (f° — p)’A7! (p° — fp) 
under constraints Af = Ap. Since f° varies in the related subspace L + VY that is parallel 
to L, and minimization is a question of projecting 6° upon L orthogonally for A! ; that is, 
along A(L*), it follows that we have 6 = f° — n*~”% Wunder asymptotic conditions. The 
random vector / is thus obtained from p°, is unbiased, and has the same covariance matrix 
as p°, so that n*—” U. 


Finally, we have: 


2 1 
E( ye p.E.) = E(p’E)* = n* iD Dele 
c 
as in the previous case. 
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Sampling Flows of Mobile Human Populations 


GRAHAM KALTON! 


ABSTRACT 


Surveys are often conducted of flows of persons, such as: visitors to museums, libraries and parks; voters; 
shoppers; hospital outpatients; tourists; international travellers; and car occupants. The sample designs 
for such surveys usually involve sampling in time and space. Methods for sampling flows of human 
populations are reviewed and illustrated. 


KEY WORDS: Mobile populations; Exit polls; Traffic surveys; Time and space sampling; Systematic 
sampling. 


1. INTRODUCTION 


Most surveys of human populations are household based, typically with a sample of 
households selected with a multi-stage sample design, and individuals sampled within the 
selected households. The household survey is a powerful method for collecting data on a wide 
range of characteristics about the population, such as social, demographic, economic and health 
characteristics and the population’s opinions and attitudes. The method is, however, not so 
effective for studying the characteristics of mobile populations. Two types of mobile popula- 
tions may be distinguished: those who do not reside regularly at a fixed location, such as nomads 
and the homeless; and members of the general population who belong to the mobile popula- 
tion under study because they are in transit, such as visitors to libraries and parks, voters at 
polling booths, shoppers, hospital outpatients, travellers, and car occupants. This paper reviews 
sample design issues for this latter type of mobile population. 

Although there are many surveys concerned with flows of mobile human populations, the 
general sampling literature contains little discussion of the sampling issues involved. The pur- 
pose of this paper is to describe the sample designs commonly adopted for surveys of flows 
of human populations, to discuss some of the special sampling issues faced, and to illustrate 
the range of applications for such surveys. The next section of the paper reviews the general 
time and space sample design used for sampling persons in transit and some of the issues 
involved in employing this design in particular situations. Section 3 then illustrates the applica- 
tion of the design in a range of different settings. Section 4 presents some concluding remarks. 


2. SAMPLING IN TIME AND SPACE 


It will be useful to consider a specific example in describing the general time and space sample 
design for sampling flows of human populations. Suppose that a survey of visitors to a summer 
sculpture exhibition in a city park is to be conducted to find out the visitors’ socio-economic 
characteristics, how they heard about the exhibition, what means of transport they used to 
get to the park, and perhaps their views of the exhibition. Suppose that the exhibition is held 
from April 1 to September 30 in the year in question, that it is open from 10 a.m. until 6 p.m. 
daily, and that there are three sites where visitors enter and leave the exhibition. 
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The sampling frame for a survey of this type is usually taken to be a list of time interval/site 
primary sampling units (PSUs). This frame is constructed by dividing the time period of the 
survey into a set of time intervals for each site. A simple construction of PSUs for the current 
example would be to divide each exhibition day at each site into two time intervals, one from 
10 a.m. until 2 p.m. and the other from 2 p.m. until 6 p.m. A more complex construction of 
PSUs could involve time intervals of different lengths on different days and/or at different 
sites. Once the PSUs are defined, a two-stage sample design is often employed. At the first 
stage a sample of PSUs is selected, and at the second stage a sample of visitors is drawn, usually 
by systematic sampling, in the sampled PSUs. 

The actual specification of the sample design for a survey of persons in transit within the 
two-stage sampling framework depends on features of the mobile population under study 
and of the survey data collection procedures. A key feature is the nature of the flow of the 
mobile population. In particular, is there a predictable variability in the rate of flow across 
PSUs? For instance, is the flow at one site higher than that at another site, or are the flows 
at some time intervals (say, Saturday afternoons) higher than those at others? Also, is the 
flow within a PSU a smooth one throughout the time interval or is it uneven, with visitors 
arriving (or leaving) in sizeable groups? Both these aspects of flow affect the sample design 
for the survey. 


If the flow is fairly uniform across the PSUs, and if the PSU time intervals are the same, 
then the number of visitors per PSU is approximately constant. In this case, the PSUs may 
be sampled with equal probabilities, and a constant subsampling fraction can be applied within 
the selected PSUs to generate an equal probability, or epsem, sample of visits. The PSUs can 
be classified in two or more dimensions (e.g. day of week, time of day, and site), and a carefully 
balanced sample across these dimensions can be obtained using lattice sampling (Yates 1981; 
Cochran 1977 and Jessen 1978). 


In many cases, the level of flow varies across the PSUs in a manner that is partly predict- 
able. For instance, the attendance at the sculpture exhibition may be known to be generally 
higher in the later shift each day and at the weekends, and particularly low on Mondays. Thus 
the PSUs comprise different numbers of visitors, that is, they are PSUs of unequal sizes. The 
usual procedure for handling PSUs of unequal sizes is to sample them with probabilities 
proportional to their sizes (PPS), or estimated sizes (PPES). In the current context, the actual 
PSU sizes are not known in advance, and estimated sizes must therefore be used. Sampling 
the PSUs with PPES works well provided that reasonable estimates of the sizes can be made. 
When PSUs are selected by PPES sampling, then the application within the selected PSUs of 
subsampling fractions that are inversely proportional to the estimated sizes of the PSUs pro- 
duces an overall epsem sample of visits. In general, an attraction of PPES sampling (with 
reasonable estimates of size) is that the subsample sizes in the PSUs do not vary greatly from 
one PSU to another. This feature is of especial value for conducting the fieldwork in surveys 
of persons in transit. When time/site PSUs are sampled by PPES sampling, lattice sampling 
cannot be applied for deep stratification. Instead, controlled selection may be employed for 
this purpose (Goodman and Kish 1950; Hess et a/. 1975). 


An important consideration in any two-stage sample design is the allocation of the sample 
between first-and second-stage units, that is, how many PSUs to select and how many elements 
to select per sampled PSU. In the case of surveys of persons in transit, that allocation is strongly 
affected by the fieldwork procedures to be used and the nature of the flow within the PSUs. 
The aim of the design is to make full use of the fieldworkers assigned to a sampled PSU while 
maintaining a probability sample of persons entering (or leaving) the site during the sampled 
time interval. 
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Many surveys of persons in transit use self-completion questionnaires, in which case the 
fieldwork process for the two-stage design described above consists of counting persons as they 
enter (or leave) the sampled site during the time interval, selecting every kth person for a 
systematic sample, and asking the selected persons to complete the questionnaire. If the flow 
is light and evenly spread throughout the time interval, one fieldworker may be able to handle 
all the tasks involved. When this is so, the sampling interval k can be chosen to give the 
fieldworker time to perform all the tasks in an unpressured way. If, however, the flow is heavy, 
either constantly or intermittently, two fieldworkers may be needed, one simply to count 
entrants (or leavers) and identify sampled persons, and the second to hand out the question- 
naires and to instruct respondents on how they should be completed and returned. With this 
fieldwork arrangement, the sampling interval can be chosen to keep the second fieldworker 
as fully occupied as possible, while making sure that he or she is able to distribute question- 
naires to all (or at least nearly all) of those sampled. Nonresponse can be a major concern with 
the self-completion mode of data collection. It is often possible to keep nonresponse to an accep- 
table level when sampled persons complete and return the questionnaire at the site. However, 
when they are handed the questionnaire with the request to complete it later and return it by 
mail, the level of nonresponse can be very high and, moreover, there is generally no way of 
following up the nonrespondents. 


When face-to-face interviewing is used for data collection, the fieldwork team for a PSU 
usually contains one counter and a small team of interviewers. The size of the interviewer team 
depends on the regularity of the flow and the length of the interview. Since persons in transit 
are likely to be unwilling to be delayed for long, interviews are necessarily mostly short. Longer 
interviews may, however, be possible if the sampled persons are in a waiting mode, such as 
waiting in line or in an airport departure lounge. The choice of sampling interval has to be 
such that there is always (or nearly always) an interviewer free to interview the next sampled 
person, and that the interviewers do not spend too much time waiting for the next sampled 
person to be selected. If the flow is irregular, allowance needs to be made to accommodate 
the peaks (for instance, the arrival of a coachload of visitors to the sculpture exhibition). 


The PPES selection of the PSUs works to equate the subsample size for each sampled PSU. 
For face-to-face interview surveys, the interviewer load is thus roughly the same for each selected 
PSU, and hence the same-sized interviewer team can be used for each PSU. A problem occurs, 
however, when the PPES measure used in selecting the PSU at the first stage is seriously in 
error. For example, a thunderstorm may substantially reduce the number of visitors to the 
sculpture exhibition on a particular Saturday afternoon, or an unforeseen holiday may substan- 
tially increase the number on another day. In the first case, applying in that PSU a sampling 
interval inversely proportional to its estimated size will leave the interviewers largely unoccupied, 
whereas in the second case it will result in a workload that the interviewers cannot handle. A 
modification that may be adopted in such cases is to change the sampling interval at the start 
of data collection to one that is more suitable for the flow actually encountered. Since this 
modification destroys the epsem property of the sample, weights are needed in the survey 
analysis. 


A general limitation to the systematic sampling of visitors at selected PSUs is that if the 
sampling interval is made long enough to enable interviewers to cope with peak flows, they 
spend much of their time without work. On the other hand, if the sampling interval is reduced, 
the interviewers are more fully occupied, but they cannot cope with peak flows. Various 
methods have been proposed to circumvent these problems (Heady 1985). One procedure is 
to take a systematic sample of times (say, every 10 minutes) and to select the next visitor to 
enter after each sampled time. This procedure might have fieldwork attractions, but it does 


186 Kalton: Sampling Flows of Mobile Human Populations 


not produce a probability sample of visitors. Persons arriving in busy periods are less likely 
to be chosen, as are those who travel in groups, and the walking habits of persons travelling 
in groups may affect the chances of selection in unknown ways. The sample generated by this 
procedure is clearly not an epsem sample. An attempt can be made to compensate for the selec- 
tion bias that operates against visitors arriving in busy times by dividing the time interval for 
selected PSUs into a set of much shorter intervals, and keeping a log of arrivals in each such 
interval. Then weighting adjustments can be employed to compensate for the variation in the 
flow across the shorter intervals. 

Another alternative procedure to systematic sampling of visitors is to take the next person 
to enter (or leave) after the last interview was completed. With this procedure, the first persons 
to arrive after gaps in the flow, perhaps the leaders of groups, clearly have greater chances 
of selection. Also interviewers may deliberately speed up or slow down their current interview 
in order to avoid or to select a particular individual. For these reasons, variants on this pro- 
cedure that select the mth person after the completed interview, where n might be set at 2, 3, 
4 or 5, have been employed. These alternatives to straightforward systematic sampling of visitors 
make more effective use of interviewers’ time, and hence enable larger samples to be obtained 
for a given fieldwork budget. However, they produce nonprobability samples, with the risk 
of selection bias that this form of sampling entails. Probability sampling provides the security 
of objective statistical inference without the need for assumptions about the sample selection 
process. With nonprobability sampling, assumptions need to be made about the way the sample 
was generated, a common assumption being that all the elements in the population have an 
equal chance of selection. Failure of the assumptions can lead to serious bias in the survey 
estimates. 

Visitors may be sampled either as they enter or as they leave a location. If data about the 
visitors’ activities in and opinions of the location are required, then leavers need to be sampled. 
In other cases, the choice between sampling entrants and leavers may depend on the nature 
of the flows. It may, for example, be difficult to sample and interview people leaving a theatre 
because they leave en masse and because they will not want to be delayed. On the other hand, 
they may be readily sampled and interviewed as they line up to enter the theatre. 

In concluding this section, attention should be drawn to the fact that the samples described 
here are samples of visits not visitors. The standard two-stage design may produce an epsem 
sample of visits, but this is not the same as an epsem sample of visitors unless each visitor 
visits the place under study (the sculpture exhibition) only once (or they all visit the same 
number of times). For most flow surveys, the visit, rather than the visitor, is the appropriate 
unit of analysis. There are, however, situations where the analytic unit is problematic. Using 
the visit as the unit of analysis, the researcher might readily accept visits to the sculpture exhibi- 
tion on two separate days as distinct visits, but might not be willing to treat two entries on the 
same day (one, perhaps, after leaving briefly for refreshments) as two visits. The use of the 
visitor as the unit of analysis presents severe problems because of the issue of multiple visits, 
and the fact that visitors will not be able to report their multiplicities. They may be able to 
recall past visits reasonably well, but they will usually be unable to forecast future visits 
accurately. 


3. SOME EXAMPLES 


This section presents some examples of surveys of flows of human populations in order to 
indicate the wide range of applications and to illustrate some of the special considerations that 
arise in particular settings. 
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3.1 A Survey of Library Use 


A survey of the use of the 18 libraries at the University of Michigan was conducted in 1984 
(Heeringa 1985). Each sampled person exiting a library was asked whether he or she had used 
the library’s materials and services during that visit. If so, the person was asked to complete 
a short self-completion questionnaire of seven questions on the materials and services used. 
Most of the 5,184 respondents completed the questionnaires on the spot and returned them 
to the survey fieldworkers; others sent them back by campus mail. A response rate of 96% 
was obtained. 

The sample design followed the two-stage time/site sample design described in Section 2. 
The survey covered the full 1984 calendar year. Each day the libraries were open was divided 
into 10 two-hour time intervals, starting at 7.30 a.m. and lasting until 3.30 a.m. the next 
morning, the two-hour interval being chosen on the grounds that it was a suitable shift for 
the fieldworkers. The PSUs were then defined to be time interval/library combinations. The 
PSUs were selected by PPES sampling, where the estimated size for a PSU was the estimated 
number of persons exiting from that library in the specified time period. Rough estimates of 
these numbers were derived from average daily usage based on November, 1983, turnstile counts 
where available, and on librarians’ estimates where not, and on an assumption that library 
exit volume was twice as high between 9.30 a.m. and 5.30 p.m. as at other times. The libraries 
were stratified into four types, and within each stratum controlled selection was employed to 
give a proportionate distribution of the sample across libraries, days of the week, and time 
intervals. 

For each selected PSU, a systematic sample of persons exiting the library was selected for 
the survey, with the sampling interval being determined to yield an overall epsem sample of 
visits. Fieldworkers were provided with a record sheet of integers from 1 up to 430, with the 
selected numbers marked on them. All they then needed to do was check off a number for each 
person exiting the library, and select the persons associated with the sample numbers. An advan- 
tage of this scheme is that fractional sampling intervals are readily handled. Where the exit 
volume for a sampled PSU was expected to be low, one fieldworker was assigned to perform 
both the counting and the contacting of sampled persons. Where the exit volume was high, 
two fieldworkers were assigned, one to count and one to contact sampled persons. There was 
also a need for more than one fieldworker for libraries with more than one exit. 


3.2 A Survey of Museum Visits 


A face-to-face interview survey of visitors leaving the National Air and Space Museum in 
Washington, D.C. was conducted from mid-July until December, 1988 (Doering and Black 
1989). The interview, which took about four to six minutes to complete, collected data on the 
sampled person’s socio-demographic background, place of residence, activities on the visit, 
exhibits of special interest, reason for visit, the size and type of group if part of a group visit, 
and mode of transport used. Children under 12 years old and persons working at the museum 
were excluded from the survey. Data were collected from 5,574 respondents, with a response 
rate of 86%. 

Each day in the survey period was divided into two half-days. Interviewing was conducted 
on one half-day every second day, alternating between mornings and afternoons. During the 
summer season, three public exits from the museum were in operation, while later in the year 
only two of them were open. During the selected half-days, survey data collection was rotated 
on an hourly basis between the exits that were open. 
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The fieldwork team for an exit at a sampled hour comprised one or two counters and two 
interviewers. The lead counter used a mechanical counter and a stop watch to keep track of 
the number of persons exiting, and to maintain a record that gave the numbers of persons exiting 
in each 10-minute interval in the hour. The lead counter also identified the persons to be inter- 
viewed. The selection of sample persons was made in order to keep the interviewers fully 
occupied. The lead counter noted when an interviewer had completed an interview and was 
ready to begin another one, and then chose the fifth person exiting after that time as the next 
sampled person. The 10-minute flow counts were used in the analysis to develop weights to 
compensate for the variation in the chance of selection associated with the variable flow of 
persons across time. 

The distinction between the ‘‘visit’’ and the ‘‘visitor’’ is particularly salient for this survey. 
Persons could, of course, visit the museum on several days throughout the survey period, and also 
could visit the museum several times on a given day. This latter possibility is particularly likely 
with the National Air and Space Museum because entry to the Museum is free, and hence there 
is no incentive to enter only once. Given this situation, it may be appropriate to define multiple 
entries on one day as a single visit for some types of analysis. For some purposes, this definition 
could be applied by restricting the analysis to those exiting for the first time on the sampled day. 


3.3 Exit Polls 


A number of major news organizations conduct polls of voters on election days in the United 
States (Levy 1983; Mitofsky 1991). Voters are sampled as they leave polling places. Those 
selected are asked to complete a short and simple self-completion questionnaire, and to deposit 
the completed questionnaire in a ballot box. A typical questionnaire contains around 25 ques- 
tions asking how the respondent voted, what the respondent’s position is on key issues, what 
opinions the respondent has on various topics, and what are the respondent’s demographic 
characteristics. Refusal rates for the CBS exit polls have averaged 25% for recent elections 
(Mitofsky and Waksberg 1989). 

The sampling of voters for election polls usually employs a straightforward two-stage sample 
design. At the first stage a stratified PPES sample of voting precincts is drawn, where the size 
measure is the number of voters in the precinct. At the second stage a systematic sample of 
voters leaving the polling place is selected, with a sampling interval chosen to produce an approx- 
imately epsem sample of voters within states. Usually only one interviewer is assigned to each 
selected precinct. The fieldwork is straightforward when a polling place has a single exit, and 
the interviewer is permitted to get close to it. When there are two or more exits, interviewers 
alternate between the exits, covering each one for set periods of time. When this applies, the 
sampling interval has to be modified accordingly. In some states interviewers are not allowed 
to approach within a certain distance of a polling place, and this can create problems if it results 
in voters departing in different directions before the interviewer can contact them. 


3.4 Ambulatory Medical Care Survey 


The U.S. National Ambulatory Medical Care Survey (NAMCS) employs a flow survey design 
to collect data on visits to physicians’ offices for physicians in office practice who direct patient 
care (Bryant and Shimizu 1988). The NAMCS has been conducted a number of times since it was 
introduced in 1973. For each survey, data collection has been spread throughout the survey’s 
calendar year in order to provide annual estimates of visit characteristics. Individual sampled 
physicians have, however, been asked to provide information for a sample of their visits occur- 
ring in only one week. The annual coverage is achieved by asking different sampled physicians 
to report on different weeks of the year. 
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The sample for the NAMCS is based on a complex three-stage design, which has varied over 
time. A broad overview of the design will serve for present purposes; for more details, the reader 
is referred to Bryant and Shimizu (1988). The first stage of the NAMCS sample design is the 
selection of a stratified PPES sample of areal PSUs, selected with probability proportional 
to population size. At the second stage, physicians are sampled from lists within the selected 
PSUs with different sampling intervals from PSU to PSU to take account of the unequal selec- 
tion probabilities for the PSUs (in the more recent surveys, different specialty classes are 
sampled at different rates). Sampled physicians are then assigned at random in a balanced way 
to one of the 52 reporting weeks of the year. Each physician is asked to record information 
for a systematic sample of his or her patient visits occurring during the sampled week, with 
the sampling interval being chosen to yield about 30 sampled visits in the week. A sampling 
interval of 1, 2, 3 or 5 is chosen for a particular physician on the basis of the number of office 
visits the physician expects during the week, and the number of days he or she expects to see 
patients. The fieldwork procedures consist of keeping a log of patient arrivals for sampling 
purposes, and then completing a short 16-item record for each sampled visit. 

The NAMCS is a survey of patient visits not patients. As such, it provides useful informa- 
tion about the nature of physicians’ work on a visit basis - the frequency of use of diagnostic 
tests, the therapies provided, and the demographic characteristics of the patients seen. It does 
not, however, provide estimates on a patient basis, such as treatments and outcomes for 
patients’ episodes of illness. 


3.5 Surveys of International Passengers 


A number of countries conduct surveys of their international travellers, both those entering 
and those leaving the country by land, sea or air. This subsection will briefly describe the sample 
designs for a survey of international air passengers conducted by the United States, for surveys 
of international air and land travellers conducted by Canada, and for a survey of international 
air and sea passengers conducted by the U.K. 

The United States Travel and Tourism Administration conducts an In-flight Survey of Inter- 
national Air Travelers to survey both foreign travellers to the U.S. and U.S. residents travelling 
abroad (see, for instance, United States Travel and Tourism Administration 1989). The survey 
is conducted through the voluntary cooperation of some thirty airlines. A stratified sample 
of scheduled flights is selected for the third week of each month and all passengers on those 
flights are included in the sample. Participating airlines are provided with a survey kit of instruc- 
tions and questionnaires in appropriate languages for each sampled flight. The airline cabin 
personnel distribute the self-completion questionnaires in boarding areas or in flight to all adult 
passengers and collect them prior to debarkation. Nonresponse is a serious problem with these 
surveys. For the 1988 survey of visitors to the United States, one half of the flight kits issued 
resulted in no returned questionnaires. For flights for which questionnaires were returned, the 
estimated response rate for non-U.S. residents was 44% and for U.S. residents it was only 20%. 

The International Travel Section of Statistics Canada conducts international travel surveys 
at both airports and landports in Canada. The surveys are undertaken in cooperation with 
Canada Customs, with customs officers being responsible for distributing the self-completion 
mail-back questionnaires. The account here is based on the report by the International Travel 
Section, Statistics Canada (1979). It reflects the survey designs that applied prior to some 
changes that have recently been made. The sample designs for the landports and airports have 
been similar, and therefore only the design for the landports will be outlined here. 

At one time the sampling scheme at landports for returning Canadian residents who had 
spent at least one night abroad was to distribute survey questionnaires to every travel party 
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on every fourth day throughout the year, the days being chosen by systematic sampling. This 
scheme proved to be unworkable because the customs officers too often failed to apply it 
correctly. It was therefore replaced by a stint scheme in which a landport was assigned two 
periods, or stints, for each quarter of the year during which the questionnaires were to be 
distributed. The stints were expected to last from 6 to 10 days, with successive stints starting 
about 6% weeks apart (Gough and Ghangurde 1977). The number of questionnaires sent to 
a landport for a particular stint was determined from the expected traffic at that port. The 
customs officers were then instructed to start the distribution of the questionnaires on a given 
day, and to continue to distribute them until none were left. This sample design is geared to 
operational limitations resulting from the use of customs officers, for whom the survey is of 
only secondary concern, as survey fieldworkers. The design has some major drawbacks, but 
perhaps a more serious concern is a response rate of 20% or less. 

The U.S. and Canadian surveys of international travellers both rely on cooperation from 
other agencies in conducting the fieldwork. This cooperation has notable benefits in costs, 
but a price is paid in terms of a lack of ability to apply rigorous controls to the fieldwork 
procedures. The U.K. surveys of air and sea travellers employ more costly face-to-face inter- 
viewing procedures. 

The 1984 U.K. International Passenger Survey included the three Heathrow terminals, 
Gatwick and Manchester airports as strata (Griffiths and Elliot 1987). Within each airport, 
days were divided into mornings and afternoons, and these periods constituted the PSUs. 
A stratified sample of PSUs was selected, and systematic samples of passengers were chosen 
in selected PSUs. A sample of PSUs for other airports was also included. Two alternative data 
collection procedures were used at seaports. At some seaports, interviewers sampled and 
interviewed passengers at the quayside. At others, the interviewers travelled on the ship, 
interviewing passengers during the voyage. In the former case, they worked shifts that cov- 
ered several sailings, and the shift became the PSU. In the latter case, the crossings were 
the PSUs. 


3.6 Surveys at Shopping Centers 


Surveys conducted at shopping centers are of two types. One type aims to describe the 
shoppers’ socio-economic characteristics, their areas of residence, and their shopping activities 
in the center. The other type uses the shopping center as a convenient location to obtain samples 
of people from the general population of the area. 

An example of a survey of the first type is a study that was conducted to examine the impact 
of the opening of a hypermarket on the outskirts of the city of Southampton, England (Wood 
1978). Surveys of shoppers were conducted in four neighboring shopping centers both before 
and after the hypermarket opened (and also at the hypermarket). At each center, the first step 
in the survey process was the enumeration of all the retail outlets and their hours of opening. 
The second step was a counting of departures of groups of shoppers from sampled shops at 
sampled hours, with counting being conducted for 15 minutes within the hour. The counting 
operation was carried out over a period of one month. Based on the counts obtained, inter- 
views were allocated between shop types and days of the week, and to specific shops and hours. 
Interviewers were then instructed to interview the given number of people leaving the shop, 
interviewing the next person to leave after they had completed the previous interview. The 
sample is one of shop visits, and shoppers could visit several shops on a particular trip to the 
shopping center. Respondents were asked about previous visits to shops in the center on this 
particular trip, and also about the number of extra shops they planned to visit. These data were 
used to develop weights for analyses of trips. 
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The second type of shopping center survey uses the selected persons at shopping centers as 
a convenience sample of the general population. Mall intercept surveys of this type are widely 
used in market research (Bush and Hair 1985; Gates and Solomon 1982). The procedures are 
often haphazard, and the samples are potentially biased. The issues involved are reviewed by 
Sudman (1980), who discusses procedures for sampling shopping centers, locations at selected 
centers, and time periods to improve the sample designs, and by Blair (1983), Dupont (1987), 
and Murry ef al. (1989). 


3.7 Road Traffic Surveys 


One form of road traffic survey relates to traffic passing through one or more locations. 
Time and space sample designs can be applied for these surveys in a relatively straightforward 
manner. Kish ef a/. (1961), for example, describe the sample design for an origin-destination 
survey of vehicles using the Port of New York Authority’s bridges over and tunnels under the 
Hudson river during 1959. A four-stage stratified PPES sample design was used for this survey. 
The PSUs were combinations of eight-hour shifts and particular bridges or tunnels. A sample 
of these PSUs was selected at the first stage, a sample of contiguous toll lanes (locations) was 
selected at the second stage within selected PSUs, a sample of specific lanes was selected at 
the third stage within selected locations, and finally a systematic sample of vehicles was selected 
at selected lanes. Interviewers stayed at one sampled location for four hours, and moved each 
hour from one traffic lane to another according to a prescribed pattern. 

Another type of road traffic survey relates to general traffic on the road. Surveys of 
occupants of passenger vehicles to study seat belt usage and drivers’ blood alcohol concentra- 
tions are of this type. A full discussion of the complex design issues involved in such surveys 
is outside the present scope; instead only a few general observations will be made. 

The method of data collection to be employed exerts a strong influence on the sampling 
procedures for a general traffic survey. Seat belt usage is mostly studied by observational 
methods, whereas the measurement of blood alcohol concentrations usually involves 
breathtesting. Shoulder belt usage of front-seat occupants can be observed in moving traffic, 
but lap belt usage and the seat belt usage of other occupants can be observed only when the 
vehicle has stopped briefly, for instance at traffic lights. Lack of street lights can preclude obser- 
vation of seat belt usage at night at some sites. Breathtesting requires the vehicle to be stopped, 
and this can be done safely only in locations where the stopped vehicle does not hinder the 
other traffic. Unlike observational surveys, interview surveys that stop vehicles face a signifi- 
cant nonresponse problem. 

An ingenious method of studying seat belt usage on interstate highways is described by Wells 
et al. (1990). For this study, an observer sat behind the driver in a passenger van that travelled 
at a slower speed than the prevailing traffic in the right hand lane of the highway. From that 
vantage point, the observer noted the shoulder belt usage of front-seat occupants of cars, light 
trucks, and vans that passed the observer’s van in the adjacent lane. 

A more usual approach to studying seat belt usage is to take observations at road intersec- 
tions and freeway exits controlled by traffic lights, and sometimes at shopping centers and 
parking lots (Ziegler 1983; Bowman and Rounds 1989). O’Day and Wolfe (1984) describe an 
observational survey of seat belt use in Michigan applying this approach. They sampled a 
number of areal units, sampled a number of intersections with traffic signals within these areas, 
sampled days for observations to be taken at these intersections, and sampled five periods of 
one hour each between 8 a.m. and 8 p.m. for observation on each selected day. Each hour 
of observation was conducted at a different intersection. The hours were selected by a scheme 
that alternated one hour working and one hour free, with the observers moving between 
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intersections in the free hours. Observations of seat belt usage were taken at the selected intersec- 
tions at the specified times for vehicles that stopped at the traffic lights. When more than one 
vehicle was stopped, observation began with the second vehicle, because of the bias associated 
with the first vehicle to stop at a light. In order to obtain more detailed information on the 
usage of child-restraints, observations were also made on vehicles entering shopping centers 
and rest areas. 

The usual approach to analyzing observational data on seat belt usage is to calculate the 
proportion wearing seat belts among those observed. Brick and Lago (1988) propose an alter- 
native measure, the proportion of estimated time front-seat occupants are belted in eligible 
vehicles to the total time in eligible vehicles. For their survey a probability sample of all roadway 
intersections, whether they had traffic signals or not, was selected. To avoid selection bias, 
observers were told the site they were to use for observation and the direction of the traffic 
to be observed in the specified 40-minute interval of observation. The time occupants were 
on the road was estimated as the length of the road segment leading to the intersection divided 
by the estimated average speed of the traffic on that segment. This estimated time was used 
as a weighting factor in the analysis. 

The sampling considerations for roadside breathtesting surveys are broadly similar to those 
for seat belt usage surveys, except that the locations for data collection need to be places where 
vehicles can be stopped safely. In the 1986 U.S. National Roadside Breathtesting Survey, local 
police officers cooperated in the survey by flagging down selected drivers and directing them 
to the survey interviewers (Wolfe 1986). The interviews lasted about 5-6 minutes. When an 
interviewer finished an interview and the respondent had taken the breath test, the interviewer 
would signal to the police officer to stop the next passing vehicle. Interviewing was conducted 
for a period of two hours at each sampled location. A count was made of all the vehicles passing 
the location in the sampled direction during the period, and the ratio of this count to the number 
of interviews conducted was used as a weighting factor in the analysis. 


4. CONCLUDING REMARKS 


As the examples in the previous section illustrate, fieldwork considerations and the economics 
of data collection play major roles in the choice of sample design for surveys of persons in 
transit. The length of the time interval used in defining the PSUs may, for instance, be dic- 
tated by the length of a suitable workshift for the fieldworkers, and this may result in PSUs 
with substantial internal variation in the rate of flow. For example, in a survey of passengers 
arriving at a railway station, a morning interviewer workshift may include a peak flow of early 
morning commuters and a low rate of flow later on. If it were not for the need to make the 
PSU time interval conform to the fieldworkers’ workshift, it would be preferable to avoid such 
variation in flow within PSUs since it leads to problems in how to subsample in the selected 
PSUs. 

When the flow of persons within a PSU is uneven, the use of systematic sampling, or any 
epsem sampling scheme, for selecting persons creates a variable workload over time. If this 
variability in workload is substantial, there are difficulties in deciding how to staff the PSU 
for the survey fieldwork, particularly for a face-to-face interview survey. The assignment of 
sufficient staff to cope with peak flows is uneconomic since interviewers will then often be 
inactive at off-peak times. Sometimes staffing for somewhat below peak flow may be prefer- 
able. This will introduce some nonresponse at times of peak flow because no interviewer is 
available to conduct an interview with some sampled persons, but it will more fully use the 
interviewers’ time. 
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The most effective use of the interviewers’ time is to assign them to interview the first person 
to arrive (or leave) after they have completed their current interview. Schemes of this type suffer 
the disadvantage of not producing probability samples, and hence there is a risk of bias in the 
survey estimates. Where cost effective probability sampling designs can be devised, they are 
to be preferred. However, the choice of a sampling scheme in which the first (or second, or 
third) person is selected after an interviewer becomes free is understandably attractive for face- 
to-face interview surveys when the flow is very variable and unpredictable. When this kind 
of scheme is employed, it is useful to take counts of the flow over short intervals of time. These 
counts may then be used to make weighting adjustments to compensate for the unequal selec- 
tion probabilities caused by the uneven flow. 
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ABSTRACT 


A sample design for the initial selection, sample rotation and updating for sub-annual business surveys 
is proposed. The sample design is a stratified clustered design, with the stratification being carried out 
on the basis of industry, geography and size. Sample rotation of the sample units is carried out under 
time-in and time-out constraints. Updating is with respect to the selection of births (new businesses), 
removal of deaths (defunct businesses) and implementation of changes in the classification variables 
used for stratification, i.e. industry, geography and size. A number of alternate estimators, including 
the simple expansion estimator and Mickey’s (1959) unbiased ratio-type estimator have been evaluated 
for this design in an empirical study under various survey conditions. The problem of variance estima- 
tion has also been considered using the Taylor linearization method and the jackknife technique. 


KEY WORDS: Continuous surveys; Sample updating; Ratio estimator; Variance estimation. 


1. INTRODUCTION 


The universe for sub-annual business surveys continually changes on account of births, 
deaths, splits, mergers, amalgamations, and classification changes. The sample design asso- 
ciated with such a universe should have the following characteristics. Firstly, it should result 
in samples which reflect the changing structure of the population. Secondly, it should distribute 
response burden by rotating units in and out of the sample. Thirdly, if there are significant 
changes in the stratification of the universe, it should be possible to redraw a new sample 
which reflects the stratification and possible changes in sampling fractions. The resulting new 
sample should have maximum overlap with the previous sample in order to minimize abrupt 
changes in the estimates and increased costs due to the introduction of new units in the sample. 
The sample design which has been proposed to satisfy these requirements is that of a simple 
random sample of randomly formed rotation groups (clusters) within each of the strata. Each 
rotation group represents either a group of units or a single unit. All units within a selected 
rotation group are selected in the sample. Rotation of the sample takes place under the constraints 
that units must stay in the sample for a certain period of time and be kept out of the sample 
for at least a certain period of time after they have rotated out of the sample. 

For given domains of interest, unbiased (or nearly unbiased) estimates are developed along 
with the associated measures of reliability (coefficients of variation). A desirable property 
of the estimation is that the estimates of domain totals should add up to the population total 
when the domains are exhaustive and non-overlapping. This can be ensured by using one set 
of weights which is independent of the domains. 

In section 2, the rotation group sampling design is developed and a number of alternative 
estimation procedures are described in section 3. In section 4, the results of an empirical study 
showing the performance of these estimators under various survey conditions are given. 
Finally, section 5 contains some concluding remarks. 


! M.A. Hidiroglou, G.H. Choudhry and P. Lavallée, Business Survey Methods Division, Statistics Canada, 11th floor 
R.H. Coats Building, Ottawa, Ontario, Canada, K1A OT6. 
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2. SAMPLING DESIGN 


2.1 Stratification and Sample Allocation 


The stratification of a business universe is usually based on one or more of the following 
characteristics: industry, geography, and size. The size measure can be univariate (e.g. sales 
or number of employees) or multivariate (e.g. revenue and assets). In our context, the primary 
strata are cross-classifications of industry and geographic regions for which estimates are 
required. Within these primary strata, secondary strata are formed using the size measure of 
the units. The secondary strata are comprised of a completely enumerated ‘‘take-all’’ stratum 
and a number of strata called ‘take-some’ strata where sampling occurs. It is necessary to have 
a take-all stratum on account of the highly skewed nature of the business universe. The take-all 
stratum boundary can be determined by a method introduced by Hidiroglou (1986). This 
method finds the optimum boundary between the take-all and the take-some strata within each 
primary stratum so as to minimize the overall sample size for a given coefficient of variation. 
The determination of this boundary also takes into account that certain units are to be sampled 
with certainty irrespective of their size. These pre-specified ‘‘take-all’’ units are units which 
are to be included in the sample on account of their complex structures. An example of a unit 
with a complex structure could be one which operates in more than one of the primary strata. 
The boundaries for the take-some strata are obtained either using the cum |f rule introduced 
by Dalenius and Hodges (1959) or the cum ix rule given by Hansen ef a/. (1953). Here xisa 
size variable available for stratification of the units in the population. 

The sample sizes for the primary strata are computed so as to satisfy planned levels of preci- 
sion for certain key estimates. The computation of these sample sizes also takes into account 
the required allocation scheme of the units to the take-some strata. It is assumed that the infor- 
mation available for computing these sample sizes is well correlated with the planned key 
variables. Given that the take-all sample units have been taken into account, the remaining 
sample is allocated to the take-some strata within the primary stratum, proportional to M4 
or X%, where M is the number of units in the take-some stratum and X is the take-some 
stratum total for the size variable being considered. The power g where 0 S gq S 1 is chosen 
according to the required allocation. Letting g = 1 results in Neyman allocation, whereas as 
q approaches zero, the resulting coefficients of variation become more equal amongst the 
different strata provided that S,,/X; does not vary significantly from stratum to stratum and 
that the finite population correction factors can be ignored. The advantages of these power 
allocations are discussed in Bankier (1988). The allocation can be adjusted to achieve the desired 
minimum sample sizes and/or maximum weights for each secondary stratum. 

The reliability criteria (in terms of coefficients of variation) can be associated with the primary 
strata in one of two ways. Either they can be specified for each primary stratum, or, for a given 
global (national) coefficient of variation (c.v.), the c.v. at the primary stratum level can be 
determined so that the c.v.’s for each industry group and geographic region are equal. An 
iterative procedure is used to determine the desired c.v.’s for each of the primary strata and 
hence the sample size within each primary stratum, so that the planned c.v.’s at the global and 
marginal levels are achieved. 


2.2 Sampling Scheme 


For each stratum, the M population units within that stratum are randomly allocated to 
a predetermined number P of population rotation groups, so that initially, the number of units 
in each of any two rotation groups differ by at most one unit. The number of rotation groups 
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is a function of sampling fractions, and time-in and time-out constraints. It may be noted that 
in order to achieve unbiasedness, the time-in and time-out constraints may sometimes have 
to be violated. A simple random (SRS) sample of p rotation groups is selected from the P 
population rotation groups. The number of rotation groups p to be selected is determined such 
that p/P is approximately equal to the desired sampling fraction f. The sample consists of all 
the units in the p selected rotation groups. Rotation of the sample occurs by acquiring an out- 
of-sample rotation group and dropping an in-sample rotation group. Births are randomly 
allocated to the P population rotation groups, one at a time, in a systematic fashion. Deaths 
are removed from the stratum only if they are detected by a source independent of the survey, 
or if they have been dead for more than a pre-specified period of time. Methods proposed by 
Kish and Scott (1971) are adopted for sample updating with re-stratification due to popula- 
tion units changing strata. The sample update maximizes the overlap between the current and 
the new samples. There are obvious advantages to redrawing the sample in this fashion. First, 
it minimizes the introduction of new units into the sample, resulting in a smoother transition 
from an operational point of view, and also minimizes cost. Second, discontinuity in the 
estimates on account of sample redraw is kept to a minimum. 


There are other sampling schemes which can be used to select the sample and rotate the units. 
These include Poisson and collocated sampling. The properties of these schemes have been 
discussed by Brewer, Early and Joyce (1972), and by Sunter (1977). Poisson sampling as defined 
by Hajék (1964) allows each unit in the population to be drawn in the sample independently 
with a given probability of inclusion. Decisions as to whether the unit is selected in the sample 
or not are made using an independent random draw or Bernoulli trial for each unit. Supposing 
that the inclusion probability of a given unit i is z;, and that a random number y; uniformly 
distributed over the interval (0,1) is generated, then the i-th unit is selected if u; < z;. This 
probability of inclusion corresponds to the sampling fraction of the stratum that the unit belongs 
to. Although the advantage of Poisson sampling lies in the simple manner in which sample 
rotation is exercised, it has certain disadvantages. Its main disadvantage is that the realized 
sample size is a random variable. This can be serious if the number of units in the stratum is 
small, possibly resulting in samples of size zero. Early and Brewer (1971) remedied this weakness 
by using a scheme known as collocated sampling. Collocated sampling is similar to Poisson 
sampling but reduces the variation in sample size by equispacing, at the cell level, the units 
over the interval (0,1). Properties of this method are provided in more detail in Brewer, Early 
and Hanif (1984). Whereas in Poisson sampling, the addition of births and removal of deaths 
do not affect the random numbers attached to existing units, the use of collocated sampling 
requires that these random numbers be slightly perturbed, possibly disturbing the rotation 
scheme by violating the time-in and time-out constraints. 


The rotation group sampling scheme has several advantages over the two previously men- 
tioned schemes. For the rotation group sampling scheme, in contrast to the Poisson scheme, 
the expected number of units on each rotation cycle is almost equal. The removal of dead units 
on a universal basis may disturb the balance of units amongst the different rotation groups. 
This can be remedied by periodically redrawing the sample with maximum overlap, keeping 
the stratification and sampling fractions unchanged. The rotation for the rotation group scheme 
can be performed without perturbing the units, thereby satisfying the time-in and time-out 
constraints. This may not necessarily be true with collocated sampling on account of the slight 
perturbations of the random numbers due to population births and deaths. These effects may 
become non-trivial over a long period of time. Another advantage of the rotation group scheme 
over the other two methods is that re-stratification and new sampling fractions can easily be 
accommodated while maximizing sample overlap. 
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2.2.1 Determination of the Number of Rotation Groups 


Assume that for a given take-some stratum, the number of population units is M and that 
the desired sampling fraction is f. Let ¢,, be the desired number of occasions a unit should stay 
in the sample. Let ¢,,,, be the minimum required number of occasions a unit must stay out of the 
sample, once it has rotated out of the sample. The required number of population rotation 
groups ‘‘P’’ and in-sample rotation groups ‘‘p’’ are determined as follows. Let, x = int 
[tin (1 — f)/f + 0.5] where int[-] denotes the integer portion of the argument. Two con- 


ditions arise: 


a) Ifx = tu, then the number of in-sample rotation groups is p = ¢,, and the number of 
population rotation groups is P = ¢&,, + x. 
b) If x < &,,, then the number of in-sample rotation groups is 


2k meen + 0.5 
basi 


and the number of population rotation groups is P = p + t,;- 
It must be noted that p/P is only approximately equal to fon account of the integer operations. 


2.2.2 Allocation of Units to Rotation Groups 


Given that at the time of initial selection, there are M population units to be allocated to 
P population rotation groups, two distinct cases arise with respect to the relative sizes of M 
and Ps Vi = Por Mi <P. 

When M = P, at least one unit can be allocated to each population rotation group. Suppose 
M=aP + @, where a > Oand ? = O are integers. In order to equalize the rotation group 
sizes as much as possible at the time of initial selection and on subsequent occasions, the 
following procedure is used. A 2 by P matrix is used to assign a rotation sequence to the units 
that will satisfy the requirements of almost equal rotation group sizes. It is used for initial sample 
selection and subsequent addition of births. The first ‘‘assignment’’ row is labelled from 1 to 
P, whereas the second ‘‘rotation’’ row is a randomized order of the first row. The correspon- 
ding rotation group numbers in the second row determine which units are in sample at any 
point in time. The M population units are assigned sequentially to the assignment rotation group 
numbers 1, 2, ..., P, the P-th unit going to the P-th assignment rotation group number. The 
(P + 1)-th unit is assigned to assignment rotation group number | and so on. This eventually 
results in having the first ‘‘?’’ assignment rotation groups with (a + 1) units and the next 
(P — ) assignment rotation groups with (a) units. The rotation group to which the M-th unit 
is assigned is termed the last assignment rotation group. This rotation group, which is assigned 
rotation group number at time of initial selection, is used for assigning future births starting 
from the next assignment rotation group number, /.e. + 1. 

When M < P, the M population units can only be allocated to a subset of M out of the 
P rotation groups. These rotation groups must be as equispaced as possible to ensure that the 
expected sample size, m = [M, will be achieved from one survey occasion to the next. For 
this case the allocation matrix is 2 by M. The first assignment row is labelled from 1 to M. 
The second rotation row is a randomization of M ‘‘z’’ numbers where 1 < z; < z; = P for 


1#jJ,1=1,...,Mand/j = 1, ..., M. The ‘*z’” numbers are created as follows. 
i) Find integers s and gq such that P = sM + gq whereqg < Mands = 0. 
ii) Generate r; (j = 1, ..., M) numbers randomly assuming the values 0 or 1, such that 


q of them have the value equal to 1 and M — q of them have the value equal to 0. 
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iii) Select a random integer ‘‘b’’ such that 1 < b < P. 


iv) Compute z; = (b + r; — 1) modP + 1l andz; = (zj_; + s + 7; — 1) modP + 1 
fOr "2.6 wi: 


v) Randomize the ‘‘z’’ numbers. Let the sequence of randomized ‘‘z’’ numbers be 
Ziy> Zin» Ac of Zing’ 


Now the M population units are assigned sequentially to the M assignment rotation group 
numbers, thereby picking up their rotation numbers. The last assignment rotation group number 
is M. Future births will be assigned starting from assignment rotation group number 1. 

It is now a simple matter to perform the basic functions of sample selection and updating. 


2.2.3 Sample Selection and Updating 


At time of initial sample selection, a given stratum will have N = min (M,P) distinct rota- 
tion groups. The units belonging to the initial sample are those whose rotation numbers are 
included in the closed sampling interval [1,0]. When M = P, the number of in-sample rotation 
groups 7 is equal to p. When M < P, the number of in-sample rotation groups 7 is approximately 
equal to fN on account of the equispacing. 

Sample rotation is carried out by shifting the sampling interval by one rotation group at 
each sampling occasion in a circular fashion. On the ¢-th occasion, units in the sample are those 
whose rotation number is contained in the interval defined as 


i) [((¢ — 1) modP + 1, (¢ + p — 2) modP + 1J, if (t — 1) modP < (P — p) 
and 
ii) [l,(p — P) + (t — 1) modP] U [(t — 1) modP + 1, Pl, otherwise. 


Effectively, rotation occurs by dropping a rotation group from in-sample and acquiring a 
rotation group from out-of-sample in a modular fashion. 

‘*Births’’ occur as a result of starting a new business activity, or a change of industrial activity 
of a unit from out-of-scope to in-scope for the survey. Births are stratified and given an assign- 
ment rotation group number within the stratum as follows. Assuming the last assignment rota- 
tion group number was ?, where 1 < ? < P, the q-th birth will be given the assignment rotation 
group number (f + g) modP. The next birth will be given the assignment rotation group 
number (f + g + 1) modP. The rotation number is then immediately obtained through the 
one-to-one correspondence between the assignment and rotation numbers. 

‘‘Deaths’’ occur as a result of the termination of business activity for in-scope units or 
changes of industrial activity from in-scope to out-of-scope to the survey. Deaths that occur 
in a take-all stratum are immediately removed from the population and sample. Deaths that 
are part of a take-some stratum are removed immediately if they are identified as such by a 
source independent of the survey process. Otherwise, they are removed after a given time period. 
This time period should be sufficiently long so that most of the population deaths would have 
been identified. Deaths in the sample and in this latter category which have not yet been removed 
are assigned a value of zero for estimation purposes. Classification values are also retained 
as such until they have been identified as changes by a source independent of the survey. 


2.2.4 Periodic Resampling 


The sampling frame changes continually due not only to births and deaths, but also due 
to changes of classification variables used in the stratification (i.e., geography, industry and 
size). These changes in the classification variables are reflected in the estimation process by 
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use of domain estimation (i.e. estimation for sub-populations). That is, the latest classification 
is assigned to data for tabulation purposes, using the original sampling weight. Over a period 
of time, changes in classification may be sufficiently important to require the examination of 
the stratification and subsequent sampling rates. One solution would be to redraw an indepen- 
dent sample, taking into account these changes, but ignoring the current sample. Such an 
approach has certain disadvantages from an operational point of view. An independent redraw 
implies that i) the newly sampled units must be initiated into the sample, ii) time-in and time-out 
constraints can be violated, and iii) the estimates may change substantially. It is therefore 
desirable to maximize the overlap between the current sample and the new sample. The 
following methodology provides such a procedure for resampling. It is an adaptation of the 
Kish and Scott (1971) method, and is based on the property that each rotation group is a simple 
random sample from the population rotation groups. 

At time of resampling, rotation will have occurred at different rates amongst the strata, 
resulting in sampling intervals with different starting and end points. Hence, assuming that 
rotation started at time ¢, and that we are currently at time f,, the number of rotations that 
have occurred isr = ft, — ¢, + 1. At time 4, the sampling interval(s) associated with a given 
stratum currently labelled as k (kK = 1, 2, ..., K) is(are) 


[(r — 1) mod + 1, (7 + pp — 2) modRh +1) if (r — 1) modRh s (FR — Px) 
and 


[1,p, — Py + (r -— 1) modP] and [(r — 1) modh + 1, R] otherwise. 


The first step associated with the resampling is to relabel the different sampling intervals, 
which have different starting points, into sampling intervals which have the same starting point. 
For the k-th stratum, the resulting sampling interval is [1, p;,]. Let b denote the starting point of 
the sampling interval at time ¢, where b is given by (r — 1) modP,; + 1. All units labelled with 
rotation number ‘‘g”’ are relabelled as (g — b + 1)ifb < g <S P,andasP, — (b — g —- 1) 
otherwise. 

The second step is to associate with each population unit currently classified to stratum k 
its new stratum ‘‘h’’. The population units of the new h-th stratum, U,, can therefore be 
expressed as the union of K non-overlapping and exhaustive sets U,,,h = 1,2, ..., L. Each 
set U,, is comprised of population units whose new stratification is # and current stratifica- 
tion is k. Some of these sets may be empty. 

The third step is to rank, on the 0 to 1 scale, sampling units within each set U,,, taking into 
account their current rotation numbers. Assume that there are M,, units in the set U,, and that 
their current rotation numbers are labelled between 1 and P;. Rank these units from 1 to M,, 
based on their associated current rotation number. Units which have the lowest rotation 
numbers are assigned the lowest ranks and units which have the highest rotation numbers are 
assigned the highest ranks. If there are any ties, these can be broken up randomly by generating 
uniform random numbers. This results in the units in set U,, to be ranked from 1 to M,,. Next, 
a unit with rank ‘‘i”’ in set U,,, 1 < i S My, is assigned a number 74; = (@,,% + i — 1)/Miy, 
where a;,; is a uniformly generated random number between 0 and 1 for each set U,, within 
U;,. These numbers represent the current rotation groups transformed to the range 0 and 1. 
Assume that the new sampling fraction associated with the new stratum is /;, and that the 
current sampling fraction is f;. If f, = f;, this implies that all units currently sampled in U;, 
will stay in the new sample and that units in the closed interval [0, f;,] will be included in the 
new sample. If f}, < f;, this implies that units must be dropped (rotated out) from the current 


Survey Methodology, December 1991 201 


sample. The units which must be dropped are those which have the lowest r;;; values. These 
represent the rotation groups which have been in the sample the longest. In order that the units 
in the new sample be contained in the closed interval [0, f,] it is necessary to relabel the 7;;;’s 
as Tn — Se — Sn) if ta = Ge — fy) and as ry; — Oe — Sn) + 1 otherwise. Assuming 
that the population units belonging to the new h-th stratum are ranked based on the ordered 
nxt S, define b,; = i/(M, + 1),i = 1,2, ..., M,. Using the b;,;’s, new rotation numbers will 
be obtained as follows. For a given new stratum h, let N, be the number of distinct rotation 
groups. Form N, disjoint intervals 


[(u el OVENS u/N;,] for u=1, ee ey 
[(M, — 1)/M, 1] for u = N,. 


The union of these intervals is the closed interval [0,1] D,,. For Dy; the new stratum A, label 
the new rotation numbers as where D,, D2, ..., Dy, where D,, < D,, for uj; < uj, 
u; = 1, ..., Ny. The i-th unit acquires rotation number D, if its corresponding },; value 
belongs to the interval J,. Assuming that all the M, units have been assigned new rotation 
numbers in this fashion, the units in sample will be those whose rotation number belongs to 
the interval [1,p,]. 


3. WEIGHTING AND ESTIMATION 


The simplest estimator which can be used in conjunction with the rotation group design 
described in Section 2.2 is the simple expansion (or simple domain) estimator. Although this 
estimator is unconditionally unbiased, it can have a large conditional bias when the rotation 
group sizes are not balanced. The removal of dead units can cause such an imbalance in the 
distribution of rotation group sizes. Other estimators which take the auxiliary rotation group 
size information into account have therefore been considered. These include the separate and 
combined ratio estimators. A drawback of the separate estimator is that its bias may accumulate 
in a non-trivial manner across strata. The combined estimator will have negligible bias, but 
possibly large variances for stratum level estimates. We have therefore evaluated the perfor- 
mance of an unbiased separate ratio estimator due to Mickey (1959). The penalty for achieving 
unbiasedness is an increase in the variance. The primary objective is to determine which of 
the above estimators is the most suitable one for the rotation group design. The criteria for 
choosing the most appropriate estimator will be based on bias and mean squared error. In order 
to simplify the comparisons, it will be assumed that each sampled unit has valid response data. 

As mentioned earlier, the h-th stratum (h = 1, 2, ..., L) is defined at some given level 
of industry, geography and size. Estimates are required for domains which can span all the 
sampling strata or be a subset of these strata. Examples of such domains are aggregations of 
variables of interest at the sub-provincial level given that the sampling may have occurred at 
a higher level, e.g. province. A desirable feature of the estimates is that the sum of any non- 
overlapping domain set must always add up to the domain defined as their union. In order 
to achieve consistency, only one set of weights can be used. 

Let y denote the characteristic of interest and y,;; be its value for the j-th unit in rotation 
group (cluster) i of stratum h. Let 6,;;(d) be an indicator variable defined as 1 if the Aij-th 
unit belongs to domain ‘‘d’’, and 0 otherwise. Then, the parameter of interest is the population 
total Y(d) given by: 
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cE Np Mhi 
¥(d) =) \° )) yw(4), 
A= == 


where Ypij(d) = Spy (@) Yni- 


As described earlier, we have a simple random sample of n, rotation groups selected without 
replacement from the N, rotation groups in the A-th stratum. Let M,; be the number of units 
in the i-th sampled rotation group within stratum h. Without loss of generality, we can assume 
that the sampled rotation groups are indexed i = 1,2, ..., m,. Let y,;(d) be the total response 
of the units belonging to domain ‘‘d”’ from the i-th sampled rotation group within stratum A, /.e. 


Mhi 
Yni(@) = 3 Ynij(@), = Eb 2, vey Mp. 
j=l 
We will consider a number of alternative estimators for the population parameter Y(d) and 
their corresponding variance. The estimators considered are of the form, 


Y,(d) = ap Wri Vni(@) , 


i=] 


where wy; is the product of the design weight and an adjustment which reflects the estimation 
procedure used. Estimators of Y(d) are obtained by aggregating over strata, that is, 


L 
Y¥(d) = i Yh (d). 
h=1 


3.1 Estimators of Total 


A. Simple Expansion Estimator 


Since the probability of selecting a rotation group in the h-th stratum is n,/N,, the design 
weight isi W, = ON, /nj} TOP i = l1PS24P 9 Hoi) oe S12 RG eo Bhesinipletexpansion 
estimator is given by 

L 
Y;(d) = yi Nn Yn (a) 5 (3.1) 
h=1 
where 


nh 
ald least 3 Yni(d) . 


i=1 


As mentioned earlier, this estimator is unconditionally unbiased, but it can have a large con- 
ditional bias. Moreover, it may not be very efficient because it does not make use of available 
auxiliary information, such as rotation group sizes. As the variation in the rotation group sizes 
may increase over time on account of removal of deaths, it may become more and more inefficient. 


B. Separate Ratio Estimator 


If the correlation between y,;(d@) and rotation group sizes M,, is large, efficiency gains can 
be realized through the separate ratio estimator defined as 


saya of) 
Ysr(d) = (TY ona (3.2) 


h=1 h 
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where 
nh 
2. ee ee 
My, = Ny Mj; 
i=1 
and 
Nh 
M, = Y\ My 


One major drawback of this estimator is that it is subject to the ratio estimation bias. Conse- 
quently, if the bias tends to be positive or negative in the majority of the strata, its accumulated 
effect can be quite significant when aggregating over the strata. 


C. Combined Ratio Estimator 


The accumulated effect of aggregation bias can be significantly reduced using a combined 
version of the ratio estimator. The combined ratio estimator is given by 


YD ka1 Nn (a) 


Y, (dq) =M 
i rk; Nam, 


(3.3) 


where M = Y4_, M,. 


D. Unbiased Ratio-type Estimator 


The bias problem caused by the ratio estimation can be completely eliminated using the 
following adjusted ratio-type estimator suggested by Mickey (1959). The Mickey estimator is 
given by 


. L Nh 
Yyvi(d) = 3 (mam + GN; —" Tick 1) | % ihe) = mta(a) |). (3.4) 
h=1 i=1 
where 
- sa Soermdeg “(j ne) 7h 
P,(d) = — PG) Ais Sea lint, = My. 
: Np » : : TAY) Mp; aoe yy i 


An undesirable feature of the Mickey estimator is that it can have weights less than one, 
including negative weights. 

For the separate and combined ratio estimators, the variances are estimated using the Taylor 
linearization method. In the case of Mickey’s estimator, a jackknife procedure is used, leaving 
out one rotation group at a time and re-computing Mickey’s estimator for the remaining 
(n;, — 1) rotation groups in the sample. Denote each jackknifed estimator as for Yuin ) 
18) ee Oe eg FS 


where 


Ce) ye wh) yui(d) 
i¥(j) 
with 


wh? = [M, — (mp, — Miyj)] (Na — my + 2) bg? + (Na Mn + 2) 
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and 
1 
bY = (m — 1)7!) YY ————_.. 
Xu (mm, — My — My) 


A jackknife variance estimator of Yayy,,(d) is given by 
Nh 
¥j(Yrn(d)) = si fig Aen B EG (d) — %,(d))’, 


where 2) (d) = Yv),(d) and Z,(d) = pth, ZV) (d). 


It can be shown that all the estimators are seater and unconditionally unbiased when 
the rotation group sizes M,; are all equal in each stratum 4. However once the rotation group 
sizes (M,;) become unequal, all estimators, except for the simple expansion and the Mickey 
estimator, are unconditionally biased. For these estimators, the magnitude of their uncondi- 
tional biases and their efficiency was assessed in a simulation study which is presented next. 


4. SIMULATION STUDY 


The purpose of this simulation was to determine which of the four estimators of aggregate 
total Y(d) and the stratum total Y,(d) would be the most ‘‘appropriate’’ for the sample 
design described in Section 2. For simplicity, the simulations were confined to a single variable 
(y), gross business income (GBI). Also, for the purpose of this simulation the domains coin- 
cided with strata. Therefore, the symbol ‘‘d’’ used to denote the domain will be omitted. 


4.1 Description of the Study 


The universe for the simulation study was defined as the set of smaller sized units belonging 
to the Wholesale Trade sector in the province of Québec for the May 1989 reference period. 
The size of each unit was based on a GBI derived from payroll deductions using a ratio model. 
Units whose GBI was below a given threshold were retained, resulting in a population of 10,953 
units. The stratification of this population was defined on the basis of Standard Industrial 
Classification at the 3 digit level. This resulted in 30 strata with a minimum stratum size of 
18 units. For each of the 30 strata, 16 rotation groups were formed by randomly assigning the 
units to the rotation groups as described in Section 2.2.2. 

For each stratum h, samples of 4 rotation groups were obtained from the 16 rotation groups 
using simple random sampling without replacement. From each stratum there were 1,820 
possible samples of size 4. Over the 30 strata there were 54,600 (30 strata times 1,820 samples 
per stratum) possible different estimates for the separate ratio estimation procedure. On the 
other hand, for the combined ratio estimator, a total of (1,820)*° different estimates could be 
produced. For the simple expansion, the separate ratio estimator, and Mickey’s estimator, all 
54,600 possible samples were drawn. For the combined ratio estimator, 100,000 samples were 
randomly drawn from the (1,820)*° possible samples. 


4.2 Evaluation Criteria 


The evaluation criteria involved bias and mean squared error. These are described next. 
For each selected sample k, an estimate Y{*) was produced for each stratum / and for each 
of the four estimators. The stratum expectation E( Y{*)) of this estimate was obtained as 
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where K is the total number of samples drawn. It should be noted that for estimators (3.1) - (3.2) 
and (3.4), E( Y;,) was in fact the true expectation since all possible samples were drawn. For 
the combined ratio estimator (3.3), it corresponded to an unbiased estimate of the expecta- 
tion. The resulting stratum bias was 


Bias( ¥;,) = E(Y,), — Yp.- 


The total bias, Bias( Y), was obtained by summing the stratum bias over all strata. 
For estimators (3.1) - (3.2) and (3.4), we have that 


Var ( YA) — 


DL 


K 
Dard Yi oeek (i ,y yt 
k= 


and 


L 
Varc? = ys Var(Y,). 
n= 


For the combined ratio estimator (3.3), we have that 
1 K 
Vane whe ihe KCE W,) 2 
(Y, agin du (Y; (Y,)) 
and 
1 K 
Vac). Yo —FCY))-. 
Wns ag Dp (f)) 
k=1 
where Y) = Yh, Yi and E(Y) = Yeni (%)- 
Finally, the stratum mean squared error, MSE(Y,), of each estimator was defined as 
MSE(Y,) = Var(¥,) + (Bias(Y¥,))? 
while the aggregate mean squared error, MSE(Y), of each estimator was given by 
MSE(Y) = Var(Y) + (Bias(Y))’. 


Four criteria were used in comparing the relative behaviour of the proposed estimators. The 
first criterion was absolute relative bias. The stratum average absolute relative bias was 


computed as 
| 


while the aggregate absolute relative bias was computed as 


| e 


Bias ( Y;,) 


ty 
A BRamiy ans 


L 


¥ Biss fp 


ARB = 
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where 
B 
= > Yy; 
h=1 


The second criterion was the ratio of absolute bias to standard error which was called 
‘absolute standard bias’’. The stratum average absolute standard bias was computed as 


ee ae ee R 
Ape Bias ( Y,) 


[vari 


while at the aggregate level, it was computed as 
ASB = | Bias(Y) | /VVar( aye 


Following Cochran (1977), a reasonable value for the maximum acceptable bias over the stan- 
dard error should not exceed 10% . Indeed, since the precision of an estimator is usually 
measured by its variance and not by its MSE, too large a bias as compared to the standard 
deviation would give a false impression of the precision of the estimator used. 

The third criterion was efficiency, defined as the ratio of the root mean squared error of 
the estimator under study, RMSE(Y¥*"), to that of the simple expansion estimator 
RMSE(Y**"). The stratum average relative efficiency was computed as 

BEF = - oH RMSE(¥j*”) /RMSE(¥;°7)}, 


tS 


while at the aggregate level, the relative efficiency was computed as 
EFF = RMSE(¥**”) /RMSE(Y**7). 
Finally, the fourth criterion was to observe the proportion of negative weights. 


4.3 Description of the Scenarios 


Four different scenarios were considered for the possible configuration of the population 
of rotation groups for the rotation group sample design described in Section 2.2. The four 
scenarios provided different combinations of the rotation group size balance (good, poor) and 
of the correlation between the rotation group sizes M,; and the survey variable y,; (good, 
scattered). In the context of rotation group balance, ‘‘good’’ means that the rotation groups 
do not differ much in size, whereas ‘‘poor’’ means that they differ significantly. In the context 
of correlation, ‘‘good’’ means that the correlation between the survey variable and the rota- 
tion group size is quite high throughout the strata, whereas ‘‘scattered’’ means that it varies 
from low to high amongst the strata. 

These scenarios represent possible configurations that will arise as the survey progresses 
through time. Scenario | reflects the survey at time of initial selection: for this case, the balance 
of rotation group sizes is good, and the correlation between rotation group sizes and the survey 
variable is good. Scenario 2 reflects the deterioration of the correlation (scattered) between 
the rotation group size and the survey variable as time progresses, due to dead units accumu- 
lating in the population. For this scenario, since the dead units have not been removed from 
the population, the balance in rotation group sizes is good, but the correlation between the survey 
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variable and the rotation group size is weakened. Scenario 3 implies that removal of the dead 
population units may result in imbalance of the rotation group sizes (poor), but strengthening 
the correlation (good) between rotation group size and the survey variable. Finally scenario 4 
represents the worst possible case, which is poor correlation between rotation group size and 
the survey variable, and poor balance in rotation group sizes. 

Scenario | was constructed by varying the rotation group sizes and leaving the GBI values 
pj unchanged for all the rotation groups. The 16 rotation group sizes were varied by sorting 
them in ascending order of y,;. Their size was set as follows. For rotation groups 1-4, 5-8, 9-12 
and 13-16, the rotation group size was set to 0.22 M,,/4, 0.24 M,/4, 0.26 M,/4 and 0.28 M,/4 
respectively. The average correlation between the GBI and the rotation group sizes was 0.86, 
ranging from 0.69 to 0.96 at the individual stratum level. The average coefficient of variation 
of the rotation group sizes was 9.2%. 

For scenario 2, the population units were randomly permuted and assigned systematically 
to one of 16 rotation groups, using the procedure described in Section 2.2.2. Approximately 
20% of the population units were then randomly assigned a y-value of zero to represent a high 
proportion of dead units. The overall correlation between the GBI and the rotation group sizes 
was 0.11, ranging from — 0.23 to 0.74 at the individual stratum level. The average coefficient 
of variation of the rotation group sizes was 4.1%. 

For scenario 3 the procedure was similar to scenario 1 except that the rotation group sizes 
differed. For rotation groups 1-4, 5-8, 9-12 and 13-16, the rotation group size was set as 0.05 
M,,/4, 0.20 M,/4, 0.30 M,,/4 and 0.45 M,,/4 respectively. The overall correlation between the 
GBI and the rotation group sizes was 0.87, ranging from 0.70 to 0.96 at the stratum level. The 
average coefficient of variation of the rotation group sizes was 60.2%. 

For scenario 4 a random rotation group size was assigned independently of the GBI values 
as follows. Suppose that for each stratum h, a, = min{M,;:i = 1, ..., N,} and b; = 
max{M,;: i = 1, ..., N,}. For each stratum h, the size M# for rotation group / was set to 7, 
€,; where e;; is uniformly distributed on the interval (a,,b,). Here 7, is a scaling factor such 
that M, = ¥ 4, Mj. The average correlation was 0, ranging from — 0.49 to 0.56 at the 
stratum level. The average coefficient of variation of the rotation group sizes was 49.2%. 


4.4 Discussion of Results 


Based on the 4 scenarios described in the previous section, simulations were performed to 
compute the absolute relative bias (ARB), the absolute standard bias(ASB), the efficiency 
(EFF), and the proportion of weights less than or equal to 0. Those quantities were computed 
for each individual stratum and at the aggregate level. The results are given in Tables 1 to 3. 
Note that all of these results are presented as percentages. 

In terms of absolute relative bias (ARB), as shown in Table 1, both the simple expansion 
and Mickey’s estimator have no bias, as expected, neither at the overall nor at the stratum level. 
The separate ratio estimator displays the most absolute relative bias while the combined ratio 
estimator displays the least relative bias. For the biased estimators, the absolute relative bias 
increases as the coefficient of variation of the rotation group sizes increases, and the correla- 
tion between the rotation group sizes and the variable of interest decreases. 

Turning to absolute standard bias (ASB), as shown in Table 2, the following observations 
can be made. The separate ratio estimator is unacceptable for most scenarios using this criterion. 
Its performance worsens as the variation in rotation group sizes increases, and as the correla- 
tion between the rotation group sizes and as the variable of interest decreases. The performance 
of the combined ratio estimator is acceptable, both at the aggregate and stratum level. 
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Table 1 
Percentage Absolute Relative Bias (ARB) 
Aggregate Level Stratum Level 
Scenario Separate Combined Separate Combined 
Ratio Ratio Ratio Ratio 
1 V2 0.07 1331 0.11 
2 0.02 0.01 0.24 0.05 
3 2.88 0.14 3.19 0.29 
4 ajay | 0.22 Wale 0.30 
Table 2 
Percentage Absolute Standard Bias (ASB) 
Aggregate Level Stratum Level 
Scenario Separate Combined Separate Separate 
Ratio Ratio Ratio Ratio 
1 13.41 0.76 a 32 0.24 
ye 0.44 0.26 0.58 0.25 
3 45.64 7208 If 12.11 0.69 
4 


43.29 1.96 9.88 0.71 


The behaviour of the estimators with respect to relative efficiency (EFF) is provided in 
Tables 3a and 3b. For Scenario 1, which represents good rotation group balance and good 
correlation, all the estimators are nearly equivalent, both at the aggregate and the stratum levels. 
For Scenario 2, which represents well balanced rotation groups and scattered correlation, the 
same conclusion holds. For Scenario 3, which represents poor rotation group balance and good 
correlation between the rotation group sizes and the survey variable, the ranking of the 
estimators at the aggregate level from highest EFF to lowest EFF is: i) the combined ratio, 
ii) the separate ratio estimator, iii) Mickey’s estimator, and iv) the simple expansion estimator. 
For Scenario 4, which represents the worst in terms of rotation group balance and correlation 
between the rotation group sizes and the survey variable, the best estimator at both the aggregate 
and stratum levels is the simple expansion estimator. The combined ratio estimate is the next 
best choice. 

Weights smaller than zero occured for the Mickey estimator in 2% of the cases. 

In conclusion, given the above four scenarios, the combined ratio estimator is a reasonable 
choice for estimation for sub-annual surveys which use the rotation group design. The simple 
expansion estimator may also be considered on account of its simplicity. However, one should 
be aware of its poor conditional properties if the rotation group sizes are not balanced. 
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Table 3a 
Percentage Relative Efficiency (EFF) at the Aggregate Level 
; Simple Separate Combined F 
Be Expansion Ratio Ratio os 
1 100.0 108.0 107.9 107.3 
Zz 100.0 100.2 99.8 100.1 
3 100.0 148.3 160.3 143.5 
4 100.0 74.3 92.3 84.3 
Table 3b 
Percentage Average Relative Efficiency (EFF) at the Stratum Level 
3 Simple Separate Combined : 
gree Expansion Ratio Ratio ny 
1 100.0 109.6 108.6 108.6 
Z 100.0 100.9 99.5 100.6 
3 100.0 183.3 180.2 174.2 
4 100.0 80.0 99.4 83.7 
CONCLUSION 


In this paper, we have presented a sample design which can accommodate the necessary 
requirements for a sub-annual business survey. These requirements have included initial sample 
selection, sample rotation and updating. Given this rotation group design, a number of 
estimation procedures have been considered and they have been evaluated via a simulation 
study. These estimation procedures are equivalent when the rotation group sizes are well 
balanced within each of the strata. In the case of unbalanced rotation group sizes, the use of 
the combined ratio estimator which used rotation group sizes as auxiliary information is 


recommended. 
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County Estimates of Wheat Production 
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ABSTRACT 


Although farm surveys carried out by the USDA are used to estimate crop production at the state and 
national levels, small area estimates at the county level are more useful for local economic decision making. 
County estimates are also in demand by companies selling fertilizers, pesticides, crop insurance, and farm 
equipment. Individual states often conduct their own surveys to provide data for county estimates of 
farm production. Typically, these state surveys are not carried out using probability sampling methods. 
An additional complication is that states impose the constraint that the sum of county estimates of crop 
production for all counties in a state be equal to the USDA estimate for that state. Thus, standard small 
area estimation procedures are not directly applicable to this problem. In this paper, we consider using 
regression models for obtaining county estimates of wheat production in Kansas. We describe a simula- 
tion study comparing the resulting estimates to those obtained using two standard small area estimators: 
the synthetic and direct estimators. We also compare several strategies for scaling the initial estimates 
so that they agree with the USDA estimate of the state production total. 


KEY WORDS: Non-probability sample; Regression; Simulation; Small area estimation. 


1. INTRODUCTION 


County estimates of farm production are more and more in demand by government agencies 
for use in local economic decision making and by companies selling fertilizers, pesticides, crop 
insurance, and farm equipment. The United States Department of Agriculture (USDA) is 
currently implementing a program to standardize and improve county estimates of farm produc- 
tion (Bass et al. 1989). County estimation programs in the past have been carried out individually 
within each state. Because of this there has been little consistency across states in data collection 
and estimation methods used to produce county estimates. The goal of the USDA program 
for county estimation is to provide a set of sampling and estimation procedures for the states 
so that county estimation programs across the United States may yield estimates of comparable 
quality. 

The new USDA county estimation program encompasses every stage of the production of 
county estimates from the construction of sampling frames through the estimation itself. The 
research described here is concerned only with the estimation of bushels of wheat produced. 
We hope, however, that our methods may prove useful in other aspects of the county estimation 
program, for example in estimating acres planted and for crops other than wheat. 

Although the county estimation procedures used in the past varied from state to state, some 
parts of the procedures were similar. A typical procedure involved obtaining initial estimates 
from the data available within each county. Then an expert would review the estimates, alter 
them in light of his personal knowledge of the farms in the sample, weather conditions, and 
other factors, and then note the implications of the adjustments on the estimated total pro- 
duction for the state. The expert might repeat this process for a number of iterations until the 
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estimates within each county seemed reasonable and the resulting state production total agreed 
with the USDA state estimate. (The USDA state total is estimated based on a large probability 
sample and is thus thought to be a more accurate estimate than the total based on the county 
estimation procedure. For this reason, states typically constrain their county estimates to sum 
to the USDA estimate.) 

Written documentation of the current county estimation procedures, as outlined above, 
typically is not available. Thus the assumptions and methods that the expert uses can not be 
inspected by others and it is practically impossible to study the procedures or replicate calcula- 
tions. In addition, one cannot obtain variance estimates or use the procedures of one state in 
another state. New methods for county estimation must address these problems. 

The data that we use in this research were collected in Kansas in 1987, before the new USDA 
county estimation sampling procedures were in use. Data from 1987 were used because the 
United States Agricultural Census was taken in that year and we may, therefore, use the Census 
data in our estimation procedure. Kansas data were chosen for use in this study because the 
county data collection program in Kansas was one of the more comprehensive programs in 
the United States. Nevertheless, the data used for county estimation in Kansas, as in most other 
states, were not collected from a probability sample of farms. Therefore, our estimation 
procedure must not require a probability sample of wheat farms. Such a procedure may also 
be useful under the new county estimation program since states still will not be required to 
choose probability samples of farms. 

There is much recent research on small area estimation (see for example Platek et a/. 1987). 
Standard small area estimation procedures, however, require known selection probabilities 
since the inverses of these probabilities are used to weight observations in standard estimators 
such as synthetic and direct estimators. (See for example Section 2 of Sarndal and Hidiroglou 
1989 for a discussion of standard small area estimators.) 

The methods considered here must be different from the usual small area estimation tech- 
niques. First, the sample of farms available to produce county estimates is not typically a pro- 
bability sample. Second, the county estimates must be constrained to sum to the 
USDA-produced state totals. Since most state agriculture departments currently do not have 
large computing facilities, an additional preliminary constraint on the estimation procedure 
is that computations must be simple enough to be performed on a personal computer. Thus, 
for our initial efforts, we prefer to avoid computationally intensive estimators such as those 
described by Fay and Herriot (1979). For these reasons, we consider a computationally simple 
estimator based on a regression model for producing county estimates of wheat production. 

In Section 2 of this paper we describe the Kansas data bases used in this study. Section 3 
presents the regression procedure for estimating wheat production while Section 4 describes 
several methods for scaling those estimates to the USDA state total. In Section 5 estimates from 
the regression models are obtained and compared to the published county estimates and to 
estimates produced using the synthetic estimator and the direct estimator. In Section 6 we 
present the results of a simulation study conducted to compare these same estimators. Section 7 
gives conclusions and areas for future research. 


2. KANSAS DATA 


For the purpose of reporting farm production, all states are divided into nine or ten districts. 
Kansas is divided into nine districts such that each of the 105 counties in Kansas is completely 
contained within one of the districts. The locations of the districts and the number of counties 
within each district are as shown below: 


Survey Methodology, December 1991 213 


District Number District Location Countries in District 
1 Northwest 8 
2, West Central 9 
3 Southwest 14 
4 North Central 11 
5 Central 11 
6 South Central 13 
fl Northeast 11 
8 East Central 14 
9 Southeast 14 


Two data bases which are used in the production of Kansas county estimates, the Planted 
Acres Data Base and the Small Grain Data Base, were available for our use in this research. 
Most of our work was done with 1987 data but we also verified our results with the 1988 data. 
The 1987 Planted Acres Data Base contains information on planted acreage for 37,094 farms 
throughout Kansas. (A farm is defined by USDA to be any place with annual sales of 
agricultural products of $1,000 or more.) Of these farms, the 22,300 that reported planting 
some wheat were used in the simulation study described in Section 5. The 1987 Small Grain 
Data Base contains production information for 5,802 farms which reported planting small grain 
crops. Of these, the 1,707 that reported planting some wheat were used in our study. 

Records on the Planted Acres Data Base are a composite of Kansas farm data from a number 
of sources collected at a number of times. First a list of names and addresses of farms is created 
using data collected by county appraisers. This data may be replaced and/or corrected using 
data from the Quarterly Agricultural Surveys and from Monthly Farm Reports. The Quar- 
terly Agricultural Surveys use stratified systematic samples of approximately 2,600 farms. The 
response rate is approximately 80%. The Monthly Farm Report is completed by about 3,000 
farmers who have agreed to file the reports. The same farmer may complete monthly reports 
for many years. The most recent data for each item appears in the Planted Acres Data Base 
and the record for any one farm in any year may contain information from a number of sources. 

The 1987 Small Grain Data Base contains information on acres planted, acres harvested, 
and bushels produced for farms responding to the Quarterly Agricultural Surveys and the 
Kansas Small Grain Survey. About 6,000 surveys were mailed to a random sample of farms 
for the 1987 Kansas Small Grain Survey; about 50% of the surveys were completed and 
returned. 

In addition to the potential problem with nonresponse bias in the Small Grain Data Base, 
there is typically a problem with response bias. The production reported by farmers is often 
lower than the actual production. The non-standard sample, nonresponse bias, and response 
bias lead us to develop the county estimation procedure described in the following sections. 


3. REGRESSION MODELING 


We propose the development of a regression model for use in producing county estimates. 
The calculations for fitting a multiple regression model can be performed using a number of 
statistical packages available for personal computers. In addition, our proposed estimator 
allows for the fact that we do not have a probability sample of farms and will produce county 
estimates that sum to the desired state total. 
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The steps in our procedure are as follows: 


1) Use multiple regression to model the relationship between farm production and some 
predictor variables using the non-probability sample of farms. 


2) Assume that the regression relationship holds for the entire population of farms in the 
state, and estimate farm production for all farms in each county. 


3) Adjust the estimates of farm production to sum to the USDA state total. 


To describe the regression model we need the following notation. For ij = 1, 2, ..., 
IU =. 105 counties. in Kansas) and fj = 14.2.0. eu a fty-let 


Nn; number of farms from i" county in sample; 


I 
i al bs n; = total sample size; 


i=1 


N; = total number of farms from i" county in population; 


= 
I 


I 
AB N; = total number of farms in population; 


i=1 


< 
II 


;; = wheat production of j'" farm in i county (in bushels); 


Xj = (1 Xj Xr... Xijp) = vector of p predictors for j farm in i‘ county. 


It is important, as we will see later, to choose predictor variables for which county totals are 
known or for which very accurate estimates of the county totals are available. The predictor 
variables must also include information related to the probability that a farm is included in 
the sample, such as a measure of the size of a farm. This will allow us to use the regression 
model to adjust for the fact that the sample is not a probability sample. 

We consider regression models of the form 


Yi; = S(Xyj| 8) + ey, 


where 8B = (8 8; B2 ... 8,) is a vector of parameters and €;; is a random error term with 
variance o*. Let the fitted values, which will be obtained using data from the Small Grain Data 
Base, be denoted by 


i 10.61 
Then the county total for the i'" county may be estimated as follows: 
Yin = YD f= DG 8), 
j=l j=l 


where a ‘‘+’’ in a subscript indicates summation over the corresponding subscript. 
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For a general form of f(X;;| 8), it would be necessary to know the value of Xj; for all farms 
in the i'" county. It is, of course, not possible to have such extensive information. If, how- 
ever, f(X;;| 8) is a linear function, then we only need to know county totals of the predictor 
variables. This is the case since, for a linear regression equation, 


N; Nj 
Visa Foye se Doe hort Br Gare gating +8 Xp! 


j=1 j=1 


BoN; + BiXin1 + BoXin2 +... + ByXizp, 


where X;,; is the total of the k" predictor for the i‘" county. 

The Y;, will be reasonable county estimates if the regression model describes the relation- 
ship between the predictor variables and production for all farms in each county as well as 
for the farms in the data base. These county estimates, however, will not necessarily sum to 
the USDA state total for production. Methods for resolving this problem will be considered 
in Section 4. 

In addition to providing county estimates of farm production, the linear regression model 
proposed above also permits us to obtain variance estimates. This is easiest to see if we write 
the county estimates in terms of matrices. Let 


x 


n X (p + 1) matrix of actual data with rows being the X;; defined above; 


Z (unknown) N x (p + 1) matrix of predictor variables for all farms in the state; 


Y = (unknown) N x 1 vector of estimates of wheat production for all N farms in state; 


B; = N X 1 column vector with elements bj; 
1 if the j" farm is in the i‘ county 
where bij = { 
0 otherwise 
A = [B, B, B; ... Bylnxr. 


The estimation procedure described above does not provide Y but instead provides a vector 
of county estimates Y¥;, = A’Y, where ‘‘T”’ indicates the transpose of a matrix. 
The variance for the county estimates is thus 


Var(¥;,) = Var(A7Y) = A’Var(Y)A = A’Var(ZB)A = 07ATZ(X™X) “'Z7A. 


Although Z itself is unknown, the product A’Z is a known matrix containing only the 
numbers of farms in a county, N,, and the county totals, X;,,, for the predictor variables. 
Thus, if we use the regression mean square error (mse) as an estimate of o”, we may obtain 
estimates of the variances of the county estimates. Variance estimates for county estimates have 
not previously been available. 
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The estimator based on a regression model as described in this section meets the requirements 
for a computationally simple estimator from a non-probability sample. In the following section 
we consider methods to adjust the estimates to sum to the USDA state totals for farm 
production. 


4. SCALING ESTIMATES TO SUM TO STATE TOTAL 


Let Y be the USDA’s estimated total wheat production for Kansas. In general, )/_, ¥;, # Y. 
Thus, we define new estimates 


7 
x 
a 


where the c; are constants such that Y/_, Y;, = ¥/-,¢; Y;, = Y. Animportant question 
is how to choose the c;. Current methods | for county estimation take c; = c (at the district 
level) and thus adjust all estimates by a common proportion. Instead, one could choose the 
c; to minimize the sum of the squared differences or relative differences between the Y;, and 
Y,,. Values of c; and Y;, for three criterion for choosing c; are given below. 


1) Choose c; = c 


If c; is taken to be a constant, then it is easy to show that 


and 


A 


2) Choose c; to minimize the sum of squared differences between Y;, and Y;, 
To choose c; to minimize the sum of the squared differences between Y,, and Y,, subject 


to 54.1 ¢¥;, = Y, we must minimize Y/4_,(¥%. — %)? = D2, (e¥4 = ¥)>with 
respect to c; using a Lagrange multiplier to impose the desired constraint. Doing this, we find 


that 
I I 
=1+](v- yf) [te Y ave] 
i=1 i=l 


I 
Yin = Yin + +|(¥- ¥ ha) [Be o/h]. 


Note that the scaled estimates, Y,,, are obtained by adjusting the original estimates, Y;,, 
by adding a factor which is a proportion of the difference between the USDA state total and 
the sum of the original county estimates. The proportion is based on the harmonic mean of 


and 
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the original estimates. Although some of these scaled estimates could be negative in theory, 
this is not considered likely in practice because farmers often underreport the amount of pro- 
duction on their farms. If the total of the original estimates exceeds the USDA state total, then 
scaled estimates corresponding to counties with small original estimates may be negative. 


3) Choose c; to minimize the sum of squared relative differences between Y;, and Y;,, 


To choose c; to minimize the sum of the squared relative differences bee Yurand ¥), 
subject.to_}),7_,¢)1;4. =-Y,-we. must minimize 5/2, 1-¥;4-—-¥j.) /%4-]7-= Diep (c-—-1)? 
with respect to c; using a Lagrange multiplier to impose the desired constraint. Doing this, we 
find that 


and 


The scaled estimates, Y,,, are again obtained by adjusting the original estimates, Y;,, by 
adding a factor which is a proportion of the difference between the USDA state total and the 
sum of the original county estimates. The proportion here is based on the squared values of 
the original estimates. As in method 2, these scaled estimates may be negative, although it is 
unlikely in practice. 

Note that we have chosen to consider the difference Y;, — Y;, relative to Y;, rather than 
to Y,,. This choice was made because in the later case the estimator, Y;,, does not have a 
closed-form solution. Thus, to meet the goal of developing computationally simple estimators, 
we chose to consider the difference Y;, — Y;, relative to Y;,. 

In the following section we will consider the effects of these three scaling methods on the 
county estimates of wheat production. 


5. COMPARISON OF ESTIMATES OF WHEAT PRODUCTION 


We used a linear regression model, as described in Section 3, to model the relationship 
between wheat production (measured in total bushels produced) and some predictor variables 
for farms in the 1987 Small Grain Data Base. The possible predictor variables that we consid- 
ered included: acres planted in wheat, acres of wheat harvested, a prediction of wheat pro- 
duction based on the 1986 county estimates, acres of irrigated wheat, acres of non-irrigated 
wheat, indicators of the district in which the farm is located, indicators of region of the state 
(east, central, west), and interaction terms. 

The most important predictor variables for the regression model were acres planted in wheat 
and some indicator of the location of the farm within the state. The variable based on the 
previous year’s county estimates did not seem to be a useful predictor for the amount of wheat 
produced on a farm in the current year. Because other possible predictor variables, such as 
irrigated acres, are not known as accurately at the county level, we decided that acres planted 
would be the single continuous predictor variable included in the model. Not all district 
indicators were needed in the regression model; that is, some districts were similar and could 
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Table 1 
Regression Models Fitted to Actual Data 
Fitted Models R2 mse 
Model 1 Bushels = —811 + 32(Pla) + 3,248/7, + 3,088/, + 2,190/, 85 5,945 
+ 2,5261, + 1,2417; — 562%, + 1,047, 
+ 399]. 
Model 2 Bushels = —281 + 28(Pla) + 1382, + 1,861J, + 2,328); 86 5,818 


+ 329%, — 3591s — 3341, — 421, + 500K, 
+11(Pla)I; + 5(Pla), +3(Pla)ly + 11(Playly 
+9(Pla)Is — 0.2(Plaig + 15(Pla)l, — 7(Pla)g 


Note: Pla is planted acres, J; is the indicator variable for the im district. 


have been grouped together. We decided, however, to include all district indicators in the model 
since groupings of districts might change from year to year or might be different for crops other 
than wheat. 

We chose to focus our study on two possible regression models: Model 1 contained acres 
planted in wheat and the district indicators while Model 2 contained these same variables and 
the interaction terms involving acres planted and the indicator variables. The models and 
measures of their fits are shown in Table 1. Although the root mean squared errors did not 
differ considerably for the two models, we felt that the difference might be magnified when 
the models were used to estimate farm production for the entire state. Thus, in the following, 
we obtain and compare estimates from both models. 

To verify that these regression models are not simply a result of some unusual feature in 
the 1987 Kansas Small Grain Data Base, we used the same set of possible predictor variables 
and searched for reasonable regression models using the 1988 data. The fits of Models 1 and 
2 to the 1988 data are similar to the 1987 fits and no other model appeared to be superior for 
fitting the 1988 data. The estimates for the parameter corresponding to acres of wheat planted 
were fairly similar in both 1987 and 1988, but the parameters corresponding to the indicator 
variables for districts showed considerable change. We believe that the indicator variables for 
districts are reflecting the effects of weather and different farming practices in different parts 
of the state. For example, irrigation is more commonly used in western and central Kansas 
than in eastern Kansas. Although farming practices are not likely to change dramatically from 
one year to the next, weather conditions may be quite different. Thus, it seems reasonable that 
the contribution of the district variable in predicting wheat production could change con- 
siderably from year to year. 

Both models were used to obtain county estimates for all 105 counties in Kansas. In Table 2, 
the unscaled estimates and their standard errors under both Models | and 2 are given for nine 
counties, one county chosen at random from within each district so that the nine counties are 
spread over the entire state. An inspection of Table 2 suggests that the estimated standard error 
for Shawnee county is an anomaly. The variance of a county estimate depends on the number 
of farms in the county, the total acres planted in wheat in the county, and the number of farms 
sampled from the district in which the county lies. District 8, in which Shawnee county is located, 
had relatively few farms in the Small Grain Data Base. The county has a moderate number 
of farms growing wheat but these farms are small in terms of acres planted. These three factors 
together result in the rather large standard error for the estimates from Shawnee county. 
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Table 2 
Regression Model Estimates for Nine Counties in Kansas 


Estimated Bushels of Wheat Produced (in thousands of bushels) 


District County Niecdiel 31 Model 2 
(no interaction terms) (with interaction terms) 

1 Decatur 4,944 4,778 
(180) (179) 

2 Trego 4,378 4,229 
(174) (188) 

3 Hodgeman 4,808 4,908 
(123) (125) 

4 Jewell 5,555 5,550 
(275) (269) 

5 Marion 5,144 4,931 
(313) (315) 

6 Comanche 2,615 2,480 
(59) (63) 

7 Leavenworth 231 262 
(53) (61) 

8 Shawnee 232 226 
(106) (104) 

9 Butler 2,374 POA) 
(331) (338) 


Note: Standard errors are given in parentheses below each estimate. 


The estimates shown in Table 2 are reasonably similar to the published county estimates 
(Kansas Agricultural Statistics 1988). While it is encouraging that our estimates are not wildly 
different from those published by Kansas, there is no theoretical basis for using the Kansas 
estimates as a standard. Thus, we carried out a simulation study to help us evaluate our 
estimators. This study is described in the following section. 


6. SIMULATION STUDY 


6.1 The Estimators to be Compared 


In the simulation study, we compared the estimates from our two regression models with 
those from two standard small area estimators: the synthetic and direct estimators. (See, for 
example, Section 2 of Sarndal and Hidiroglou (1989) for a discussion of standard small area 
estimators, including the synthetic and direct estimators.) The synthetic estimates are obtained 
by allocating the state total for wheat production to the counties according to the proportion 
of total acres planted in wheat within each county. The direct estimates are obtained using only 
the sampled farms in a county to estimate wheat production for that county. 

We expect the synthetic estimates to have a large amount of bias because counties in dif- 
ferent parts of the state have different farming practices and different weather conditions, while 
the synthetic estimator treats each county as if it were representative of the entire state. The 
synthetic estimates, however, will have relatively small variances because they are obtained 
using all the data from the entire state. 
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Since the direct estimate for a county is based only on the sample data within that county, 
it will have a relatively large variance but it should have smaller bias than the synthetic estimate. 
At least one farm from a county must appear in the sample to make it possible to obtain an 
estimate for that county, and at least two farms are needed in the sample to make variance 
estimation possible. In the 1987 Kansas Small Grain Data Base, three counties had no wheat 
farms in the sample and three additional counties had only a single farm in the sample. Although 
we are comparing our regression model estimates to the synthetic and direct estimates, it should 
be noted that the latter two estimators require that the data be from a probability sample. This 
requirement is not met by the Kansas data. 


Table 3 
Numbers of Farms and Production Levels by District and Planted Acres 


Planted Acres in Farm 


District a meen ser Si ee Pea eh) Lh a le Ge 

0-99 100-249 250-499 500-999 > 1,000 
1 M# 354 638 531 302 85 
m;* 27 45 51 40 9 
bu/pa* 34.68 37.18 37.76 39.21 38.68 
2 M, 266 550 572 377 161 
m; 27 49 47 55 33 
bu/pa 35.92 33.62 36.78 39.09 34.85 
3 M, 264 549 610 537 264 
m; 31 80 16 98 61 
bu/pa 26.93 32.84 35.03 36.79 33.13 
4 M, 956 939 626 271 50 
mM; 62 37 23 21 7 
bu/pa 36.81 36.91 39.70 39.87 39.41 
5 M; 1,236 1,529 912 350 54 
mM; 92 93 51 26 3 
bu/pa 31.79 32.25 31.69 36.85 33.65 
6 M; 1,181 1,427 1,160 793 249 
mM; 96 96 81 55 20 
bu/pa 26.24 26.88 28.78 27.87 26.72 
7 M, 957 242 67 9 3 
mM; 62 5 2 0 0 

bu/pa 33.87 40.81** 40.81** 40.81** 40.81** 
8 M; 1,126 251 52 9 1 
m; 56 11 2 0 0 

bu/pa 26.02 11.48** 11.48** 11.48** 11.48** 
9 M; 1,122 431 166 59 12 
m; 47 19 7 3 1 

bu/pa 23.57 23.87 27.63** 27.63** 27.63** 


* 


M; is the number of farms on the Planted Acres Data Base, m; is the number of farms in the Production Data 
Base, and bu/pa is the ratio of bushels produced to acres planted. 
** Cells of this district were grouped to obtain bu/pa values. 
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6.2 The Simulated Population and Samples 


We first simulated a population of wheat farms by generating production values for all 22,300 
farms reporting acres planted in wheat on the Planted Acres Data Base. Because production 
rates appear to vary by district and size of farm (see Table 3), we generated bushels-per-planted- 
acres (bu/pa) from 37 different distributions. These distributions were based on the bu/pa data 
from the Small Grain Data Base. (Notice that in the eastern districts of Kansas, districts 7, 
8 and 9, there were few or no sampled farms in several size-of-farm classifications. Those 
classifications were grouped as indicated in Table 3 for the purpose of simulating bu/pa values.) 
Histograms of the observed bu/pa from the Small Grain Data Base were generated by district 
and five sizes of farm: 0-99, 100-249, 250-499, 500-999, and 1000 or more acres of wheat 
planted. Since these histograms generally appeared mound-shaped, we chose to use normal 
distributions to model the distributions of the bu/pa. The means and variances of the normal 
distributions were taken to be the sample means and variances of bu/pa from the wheat farms 
in the Small Grain Data Base within the 37 district by size-of-farm classifications. 

After the bu/pa values were generated from the appropriate normal distributions for each 
farm, the bushels of wheat produced were obtained by multiplying the simulated bu/pa by 
the reported acres planted in wheat for each farm. Ten samples were generated from the 
resulting simulated population. Since there was no sampling design to follow in creating these 
samples, we sampled each farm within the district by size-of-farm classifications with pro- 
babilities equal to the observed frequencies with which farms on the Planted Acres Data Base 
appeared in the Small Grain Data Base. That is, farms within classification C, say, were chosen 
to be in the sample with probability equal to 


Number of farms in classification C in Small Grain Data Base 


Number of farms in classification C in Planted Acres Data Base 


Our goal in using such a sampling scheme was to make the simulated samples as similar as 
possible to actual samples even though we do not know what the selection probabilities for 
the actual samples were. 


6.3 Comparison of the Four County Estimators 


We used the four county estimators (the two regression, the synthetic, and the direct) to 
obtain wheat production estimates for all 105 counties from each of the ten simulated samples. 
The resulting estimates were then compared to the ‘‘true’’ production values obtained for each 
county from the simulated population. This comparison allows us to evaluate the amounts 
of bias and variability in the estimates for each county. Figure 1 presents the values of all four 
estimates from each of the ten samples along with the true production values for the nine- 
randomly chosen counties, one from each district, which were previously mentioned in 
Section 5. 

As expected, the synthetic estimates exhibit considerable bias. Indeed, only in district 2 does 
the range of estimates include the true population value. The ranges of the direct estimates 
are all larger than those of the synthetic estimates but those ranges do include the population 
values. The ranges of estimates from the regression models appear to be less than those of the 
direct estimates. For about half of the counties pictured in Figure 1, the estimates from Model 
1 appear to exhibit some bias. The estimates under Model 2 seem to exhibit less bias. On the 
basis of this comparison of estimators we prefer Model 2, the regression model with the interac- 
tion terms. 
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Synthetic Estimates Direct Estimates 


Yield in 1,000 Bushels Yield in 1,000 Bushels 


1,000 


0 
es 4 he" 6 18. 8s a y babmtrayie. “i somata. Tint» ital thay 9 el a hee 
District Number District Number 
Model 1 Estimates Model 2 Estimates 
(no interaction terms) (with interaction terms) 
Yield in 1,000 Bushels Yield in 1,000 Bushels 


1,000 


kOe a 4 ee Of ome 2. Oo @ EO. OC oe oe oe 


District Number District Number 


Note: Estimates are for one county chosen at random from within each district. 
© = estimate from one of the ten simulated samples, 
+ = true value from simulated population. 


Figure 1. Comparison of Estimators for Nine Counties 
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6.4 Comparison of the Scaling Methods 


The same four sets of estimates for all counties from the ten sets of simulated samples were 
next scaled to agree with the state total from the simulated population using the three scaling 
methods described in Section 4. The resulting scaled estimates were compared to the true county 
production values for the simulated population. The comparison was made using the mean 
of the absolute value of relative error which is defined as follows: 


I 
GD] Cie NO /¥A 


i=1 


Figure 2 shows the values for all ten samples of the mean over the 105 counties of absolute 
relative error. This error is given for all four estimators under no scaling and under each of 
the three methods of scaling. 

From Figure 2A, we see that the scaling method which minimizes the sum of squared dif- 
ferences produces very poor final estimates; the average of absolute relative differences between 
the final estimates and the county production values for the simulated population is quite large 
compared to that of the other scaling methods. This large error results from the fact that the 
total wheat production in one county may be quite different from that in another county. Since 
the scaling procedure minimizes the squared differences between the original and the final 
estimates, a county with a very small original estimate may have a final estimate that is changed 
considerably relative to the original estimate. These large changes in estimates do not seem 
warranted; hence we drop this method of scaling from consideration. 

Figure 2B, a refinement of Figure 2A, provides a more detailed comparison of the four 
estimators under no scaling and under the two remaining scaling methods. We see from this 
figure that the error is generally smallest for the regression model with the interaction terms. 
This supports our choice of Model 2 in the previous subsection. In addition, Figure 2B sug- 
gests that there is little difference between the original unscaled estimates and the final estimates 
under either scaling method. In fact, the total of the original county estimates is not far from 
the simulated population total. Thus, the scaling constants, c;, are all quite close to one. Since 
the two methods of scaling produce similar estimates, there is no reason to use the more diffi- 
cult scaling method; the constant scaling method may be used. 


7. CONCLUSIONS AND AREAS FOR FUTURE RESEARCH 


We have shown that a regression model may be used to obtain reasonable county estimates 
of wheat production. The model we selected used acres planted, district indicators, and interac- 
tion terms as predictor variables. The regression model does not require a probability sample 
of farms and it does permit the estimation of variances of the county estimates. The estimates 
based on the regression model may be scaled to agree with state total production using a constant 
scaling factor since the alternative scaling method did not produce markedly different county 
estimates. 

Many areas for future research in county estimation of farm production remain. For 
example, the county estimates from our simulation study suggested that the inclusion or exclu- 
sion of large farms (1,000 or more acres of wheat planted) from the sample for a district could 
have a large effect on the estimates for counties in that district. This was particularly true for 


224 Stasny, Goel and Rumsey: County Estimates of Wheat Production 


Mean Absolute Relative Error 
30 


25 


20 


15 


10 


No Scaling Constant Scaling Min. Sq. Diff. Min. Sq. Rel. Diff. 


Mean Absolute Relative Error 


0.20 


0.15 


0.10 


No Scaling Constant Scaling Min. Sq. Rel. Diff. 


Note: Data are from the simulated samples. 
N = Model 1 Estimator (no interaction terms), 
X = Model 2 Estimator (with interaction terms), 
S = Synthetic Estimator, 
D = Direct Estimator. 


Figure 2. Comparison of Estimators and Scaling Methods 
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districts which had few of these larger farms. Since large farms most likely account for a sizable 
proportion of farm production, it might be worthwhile to handle large farms separately in a 
county estimation procedure. States might also consider altering their sampling plans so that 
the largest farms are included in the samples with certainty. 

Additional work is needed to determine whether a regression model similar to that devel- 
oped for wheat is appropriate for other crops as well. In particular, it would be useful to discover 
if such models can be used for rare crops where there is much less available data. We should 
also note that the similarity in the state total and the total of the county estimates, which was 
observed for the actual data as well as for the simulated samples, may be characteristic of wheat 
production but not of all crop production. Future research should consider whether other crops 
require a scaling method other than constant scaling. 

We chose to begin our research on the county estimation problem by studying methods of 
estimating production. An additional problem for future research is the estimation of total 
acreage planted for various crops. In this research we used 1987 agricultural census data to 
provide the needed information on numbers of farms and acres planted in wheat within each 
county. The agricultural census, however, is taken only every five years. In the intermediate 
years, changes in numbers of farms and acres planted must be estimated from sample data. 
We expect such changes in census values to be small for major crops like wheat in Kansas, but 
we anticipate greater difficulty estimating these quantities for less common crops. 

Finally, the requirement for a computationally simple estimator, which led us to propose 
an estimator based on a regression model, may no longer be necessary as state agricultural 
offices are being linked to a large, national computer system. Thus, in our future research on 
county estimates of farm production, we plan to consider more computationally intensive small- 
area estimators. 
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ABSTRACT 


The Canadian General Social Survey is an annual survey that aims to provide data on the demographic 
and social characteristics of Canadians. This paper provides an overview of the program, based on the 
experience of the first five surveys. The objectives of the program, the methodology used, the themes 
and issues addressed, the program outputs and the plans for the future are all discussed. 


KEY WORDS: Social surveys; Telephone surveys; Random digit dialing; Time use surveys; Health 
surveys. 


1. INTRODUCTION 


Statistics Canada’s social statistics program is concerned with providing information on 
the demographic and social characteristics and conditions of Canadians. The program’s output 
sustains the development of policy on many critical social issues. 

The Census of Population, held every five years, is the cornerstone of the social statistics 
program, providing benchmark information on the demographic, social, and economic condi- 
tions of the population and the basis for future sample surveys of the population. In addition 
to the Census, activities include on-going surveys and other statistical programs, many based 
on administrative data sources, in the areas of Health, Education, Culture, Justice, Public 
Finance, Employment and Unemployment, Income and Expenditures and Demography. 

While household surveys have long been an important part of the social statistics program, 
the regular survey program has historically been directed mainly at labour market and income 
related issues and there have been no regular ongoing surveys in areas such as health, education, 
justice or culture. In order to partially fill this data gap Statistics Canada established in 1985 
a General Social Survey (GSS) program. 

The purpose of this paper is to outline the nature and scope of the GSS program and to 
describe its evolution over the past five years. Included is a description of the methodology 
and the content of the five surveys that make up the program. Finally there is a brief discus- 
sion of some future directions for the program. 


2. GSS PROGRAM OBJECTIVES AND STRUCTURE 


The period 1930-1980 witnessed a rapid rise in the number and size of social programs in 
Canada. Whereas in the early 1930’s all government expenditures on social programs accounted 
for about 10% of GNP, by the early 1980’s this expenditure had climbed to about 30%. Along 
with this rise came an increased demand for and use of data and information to monitor and 
analyze social trends, and over the years, Statistics Canada expanded its social statistics program 
to meet growing requirements. Nonetheless, the more extensive use of available data in recent 
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years revealed major areas of weakness where relevant data were too narrow and restrictive 
for the effective planning of policy programs, products, and services, or for determining the 
allocation of resources between competing alternatives. 

In the early 1980’s, a shortcoming of the social statistics program was that aside from the 
labour market and income areas, most other social data were derived from administrative 
records or surveys of institutions. These data sources provided only limited information on 
the population who came in contact with social institutions and no data on the need for, or 
impact of, social programs on the general population. Such data can only be obtained through 
a general population survey. 

While a case could have been made for regular, frequent and large scale surveys in a variety 
of fields (eg. health, education, criminal victimization), resources to mount such a large scale 
program were not available. Instead Statistics Canada initiated a much more modest annual 
General Social Survey which over five years would cover major topics of importance and which 
would in the long term serve as a vehicle for monitoring social change. In the short term it could 
also serve as a vehicle to collect limited data on topics of current social policy interest. The 
total annual budget for the GSS was originally set at about one million dollars (CAN) and the 
program was funded by an internal reallocation of Statistics Canada resources derived from 
efficiency gains in the Labour Force Survey program. 


The objectives of the GSS program are two-fold: 

e To gather data with a degree of regularity on a broad range of social trends in order to monitor 
temporal changes in the living conditions and well-being of Canadians; and, 

¢ To provide information on specific social policy issues of current or emerging interest. 


To meet these objectives, the GSS program was established with an annual survey cycle. 
In order to cover the wide range of social issues for which data are required, the GSS program 
consists of five survey cycles, each covering a different core topic. The collection of data for 
these topics is thus repeated every five years. The core topics identified for the five cycles are: 
1. Health 
2. Time Use 
3. Personal Risk (accidents and criminal victimizations) 

4. Education and Work 
5. Family and Friends. 


An additional objective in planning content was to include questions that would be useful 
in deriving indicators of the quality of life, for example, measures of life satisfaction, attitudes, 
perceptions, or beliefs. 
The content of a GSS cycle consists of the following three modules: 
¢ Core content, which is repeated every five years in order to gather information to monitor 
trends in living conditions and well-being. 
¢ Focus content, which varies from survey to survey and is aimed at the second survey objec- 
tive of providing information on specific policy issues of particular interest to certain federal 
departments or policy groups. 

¢ Classification content, which is collected in every cycle and consists of a set of basic 
demographic and socio-economic variables that enable the delineation of various population 
groups to facilitate the analysis of core and focus content. 


While core and classification content are funded by Statistics Canada, costs associated with 
focus content are recovered from sponsors. 

The target population for the GSS consists of the non-institutionalized population aged 15 
and over living in the ten provinces. It was decided that the Labour Force Survey would not 
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be used as a vehicle for the GSS in order to avoid placing an excessive response burden on LFS 
respondents and to allow the GSS to use sampling and collection methodologies and sample 
allocations that differ from those of the LFS. The target sample size for each cycle is 10,000 
individuals which was arrived at as a compromise between the competing demands of preci- 
sion of estimates, budget and length of interview. However, there exists within the GSS program 
the potential for sponsors to expand the sample for a target population or geographic area. 
The first survey on health was conducted in late 1985 and the other surveys followed at approx- 
imately one year intervals. The fifth cycle on the family was conducted in early 1990 and data 
collection for Cycle 6 began in January 1991. 

The themes and research issues which are covered by each of the surveys are discussed in 
more detail below. However, before considering these the methodology of the survey is exam- 
ined in more detail. 


3. METHODOLOGY 


3.1 Requirements and Constraints 


The following are the principal methodological requirements of the GSS: 1) it should allow 
for extensive analysis of the adult Canadian population at a national level and somewhat less 
detailed analysis at the regional level (this has implications on both the sample size and on the 
amount of data collected from each respondent); ii) it should have an acceptable cost; iii) it 
should have a design appropriate for a multipurpose survey; and iv) it should provide public 
use microdata sets that could be used for analysis by researchers outside Statistics Canada 
without too much difficulty. 

These requirements all interact with the choice of data collection mode, sample design and 
sample size, but the last two were principally responsible for the choice of sample design, while 
the sample design and the first two requirements were largely responsible for the choice of data 
collection mode and sample size. 

The last requirement suggests that the sample design be simple as the design information 
that would be necessary to analyse complex survey data cannot generally be made available 
on public use files. Requirement ii1) suggests that the design not be highly optimized for specific 
variables. 


3.2 Mode of Data Collection 


The choice of data collection mode involved balancing a number of competing factors: cost 
per interview, length of interview, response rate, accuracy of information collected and sample 
size. The level of detail required in the data collected meant that interviews were expected to 
last 20 to 30 minutes per respondent. To reduce response burden at the household level and 
to avoid a cluster effect at the household level it was decided that only one person per household 
would be interviewed. The principal data collection methods considered for the survey were: 
self-completed mail-back questionnaire; personal interview; and telephone interview. The high 
non-response rates experienced with self-completed mail-back questionnaires were felt to be 
unacceptable (in terms of potential biases) given the heterogeneity of the target population. 
Personal interviews were felt to offer a number of advantages that would improve the quality 
of the data collected such as low non-response rates and low item non-response rates, but 
suffered from the disadvantage of high cost. In addition, many designs used to reduce the cost 
of personal interviewing have multiple stages of selection and are highly optimized for a few 
variables. (To not use a design and frame currently used for personal interviewing would have 
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been unreasonably costly.) These complicated designs make analysis of the resulting datasets 
difficult and the optimization leads to high design effects for some variables. These high design 
effects make such designs less appropriate for multipurpose surveys like the GSS. Experience 
with telephone surveys at Statistics Canada indicated that fairly high response rates could 
be achieved at reasonable cost. In addition, random digit dialing (RDD) sampling methods 
allow the efficient selection of samples that are simple random samples or nearly simple 
random samples. 

For these reasons, the GSS has used telephone sampling (RDD) methods and telephone inter- 
viewing for most of its sample in all cycles conducted to date. When there has been a need to 
focus on special target groups its main sample has been supplemented with individuals selected 
from list frames. In Cycle 1 it was felt that face to face interviews should be used for many 
of the interviews with elderly respondents. 


3.3 Target Population 


The target population of the GSS is all persons over the age of 14 permanently living in 
Canada, with the following two exclusions: i) residents of the Yukon and Northwest Territories, 
and ii) residents of institutions. This target population is different from that of the Labour 
Force Survey, which in addition excludes residents of Indian Reserves and full-time members 
of the Canadian Armed Forces. 


3.4 Sampled Population 


The sampling methods used for the GSS exclude some members of the target population 
from the sample. During weighting, these exclusions are implicitly assumed to be similar to 
the sampled population (missing at random) and the final weights produce estimates for the 
target population. 

When telephone interviewing methods are used, those persons living in households without 
telephones are excluded from the sample. This affects less than 2% of Canadian households 
covered by the Labour Force Survey (Statistics Canada 1989, 1990b). This high rate of tele- 
phone penetration is not uniform across age and income groups and varies from province to 
province: 95.4% of households in the province of Prince Edward Island have telephones while 
99.2% of those in the province of Ontario do; 99.1% of households with incomes between 
20 and 25 thousand dollars have telephones while only 93.9% of those with incomes less 
than 10 thousand do. Some subpopulations have much lower rates of telephone ownership 
than the average; for instance, only 86.7% of low income persons under 65 living alone have 
telephones. 

The GSS does not in general accept proxy responses and so individuals who cannot use a 
telephone (those unable to hear or unable to speak) or who cannot be reached by phone during 
the survey period or who do not speak either English or French are excluded from the respon- 
ding population. (For the sixth GSS cycle (on the health of Canadians) it was decided to accept 
proxy responses in those situations where the selected respondent could not complete the inter- 
view due to a health problem.) 

When supplementary samples are drawn from lists of households interviewed by the Labour 
Force Survey (as was done for GSS Cycles 1, 5 and 6), residents of Indian Reserves and full- 
time members of the Canadian Armed Forces are excluded from these samples. These exclu- 
sions represent less than 0.5% of the population over the age of 65. (This is the only age group 
that has been sampled this way.) 
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3.5 Stratification 


The stratification used by the GSS is determined by estimation requirements, operational 
requirements, restrictions on the definition of strata imposed by RDD sampling, weighting 
problems specific to RDD sampling and the special needs of sponsors. Since some estimates 
are required at the provincial level the GSS strata never cross provincial boundaries. For oper- 
ational reasons a stratum must be interviewed from a single Regional Office, thus strata never 
cross Regional Office boundaries. The RDD sampling method used requires that strata be defined 
as aggregations of telephone exchanges. During weighting, accurate estimates of the sizes of 
strata are needed, thus the strata (defined on the basis of telephone geography) need to correspond 
closely to aggregations of units for which accurate population data or estimates were available. 
These accurate data are available in intercensal years at the Census Metropolitan Area (CMA) level. 

The basic stratification based on these requirements starts with the provincial boundaries 
as stratum boundaries. In Cycles 1 to 5, Saskatchewan and Ontario were each covered by two 
Regional Offices, so they were both divided in two by a stratum boundary. Further, within 
each of the areas thus obtained, the CMA’s formed a stratum and the non-CMaA areas another 
stratum. In addition, the two largest CMA’s, Montreal and Toronto, were each separate strata. 
For Cycles 1 to 5 this gives us a total of 25 strata: one in Prince Edward Island (there is no 
CMA in PEI), two in each of Newfoundland, Nova Scotia, New Brunswick, Manitoba, Alberta 
and British Columbia, three in Quebec, four in Saskatchewan and five in Ontario. For Cycle 6 
there were 21 strata: one in Prince Edward Island, two in each of Newfoundland, Nova Scotia, 
New Brunswick, Manitoba, Saskatchewan, Alberta and British Columbia, and three in Quebec 
and Ontario. 

This is the basic stratification used by the GSS, but modifications to this basic common 
stratification to accommodate the particular needs of the subject matter or of the sponsors 
are possible and have been implemented. In Cycle 2 the special interest in language use indicated 
that separate strata with higher sampling fractions should be used in ‘‘contact regions”’ in which 
there were thought to be large numbers of both anglophones and francophones. In Cycle 5 
the interest of a client in producing estimates for certain sub-provincial regions of Ontario led 
to the definition of a special stratification. 


3.6 Allocation 


The target sample size of the GSS is 10,000 completed interviews. This sample has been 
allocated to provinces in proportion to the square roots of their population sizes. The allocation 
to strata within provinces has been in proportion to their sizes. The square root allocation is a 
method of increasing the sample sizes for the smaller provinces (when compared with a propor- 
tional allocation) without compromising the precision of Canada level estimates as much as 
an equal allocation. The method of Kish (1976) for arriving at an allocation that explicitly 
balances the need for provincial and Canada level precision has been investigated, but the 
resulting allocations yield little improvement in the precision at the Canada level while changing 
the allocations to some provinces dramatically and in a way felt to be undesirable. 


3.7 Telephone Sampling Method 


Except for supplemental samples of the population over 65 selected using lists of households 
interviewed for the Labour Force Survey, the GSS samples have been selected using random 
digit dialing methods. Two methods of sample selection have been used, the Waksberg (1978) 
method and the elimination of non-working banks method. Both methods use information 
obtained from telephone companies to improve the success rate of reaching households. The 
choice of methods depends on the level of detail of the information available. 
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Telephone numbers in Canada are ten digit numbers that can be decomposed into a three digit 
‘*Area Code’’, a three digit ‘‘Prefix’’, and two two digit fields, the first of which we refer to 
as a bank identifier. Thus within each ‘‘Area Code-Prefix’’ (ACP) there are ten thousand possible 
numbers and within each ‘‘Area Code-Prefix-Bank’’ (ACP-Bank or simply bank) there are one 
hundred possible numbers. For example, here is a fictitious telephone number and its components: 


216-357-4675 


216 Area Code 

ey Prefix (exchange) 
46 Bank Identifier 

a3 Number 

216-357 ACP 

216-357-46 ACP-Bank (bank). 


When the only information that is available is a list of ACP’s, the GSS uses the Waksberg 
method of generating the sample. In this method, banks are selected with probability propor- 
tional to size, where the size measure is the number of residential telephone numbers in the 
bank. Within each selected bank a simple random sample of residential numbers is selected. 
When the sample size is the same in each bank, this method yields an equal probability sample 
of residential telephone numbers. The sample size within banks used by the GSS has been 6. 
This method has the advantage of improving the success rate of selecting residential numbers 
with the disadvantage of producing a clustered sample. For instance, in some rural areas of 
western Canada only approximately 6% of the numbers generated using lists of ACP’s are 
residential, while the success rate during the second stage of seletion is about 50%. The design 
effects due to clustering are small for many variables, on the order of 1.0-1.3. 

When more detailed information is available that allows the creation of a list of banks con- 
taining one or more residential numbers (‘‘working banks’’) the method which we call the elimination 
of non-working banks method (ENWB) is used. A simple random sample of numbers within 
the working banks is selected and non-residential numbers are rejected, yielding a simple 
random sample of residential numbers. Since the first GSS in 1985, sampling has shifted more 
and more to the ENWB method as more information has become available from the telephone 
companies. For Cycle 6 (conducted in 1991) the ENWB method was used for the entire sample. 

A system of computer programmes for the Regional Offices of Statistics Canada has been 
written to implement these two sampling schemes and to monitor the progress of the survey. 
Within a stratum, the entire sample must be generated using the same sampling method. 

After a household has been reached by telephone, a list of the names and ages of all 
household members is collected and, using this list and a set of random numbers printed for 
each questionnaire, one person 15 years of age or older in the household is selected to be inter- 
viewed. This is the method of Kish (1949). 


3.8 Special Samples 


Sponsors of the GSS have the opportunity to fund additional interviews. These additional 
samples can be simple increases in the RDD sample size for one or more strata or they can be 
drawn from other sampling frames. 

In Cycles 2 and 5 the RDD samples in strata of special interest to sponsors were increased. 

In Cycles 1, 5 and 6 additional samples of special interest groups were used to supplement 
the RDD sample. In these cases samples of persons aged 65 and over were selected using lists 
of households that had recently been part of the LFS sample. 
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3.9 Response Rates 


One disadvantage of telephone surveys is that respondents seem to find it easier to refuse 
to participate in a telephone survey than in a survey with personal interviews. Telephone 
soliciting is being used regularly by businesses to sell products and services and everyone has 
to learn to say no over the phone. In addition, new technologies such as answering machines 
and special features being added to telephone systems are making it possible and easy for people 
to screen their incoming calls. 

Table 1 gives response rates for the first five cycles of the GSS. The categories ‘‘Other 
Household Non-Response’”’ and ‘‘Other Respondent Non-Response’’ include non-interviews 
due to language problems, illness, death in the family and absence for the survey period; some 
of these non-responses are undoubtedly refusals in disguise. In all cycles except Cycle 2 inter- 
views were conducted as soon as possible after contacting the households. In Cycle 2 there was 
a gap of about a month between the initial contact with the households and the interviewing; 
there is a component of non-response that can be directly attributed to this time lag. From 
the table it seems that there may be a trend toward lower response rates over the five cycles. 

If we consult Table 2, which presents response rates for individual Regional Offices for Cycles 3 
to 6, we see that the situation is not so simple, with many offices (Halifax, Montreal, Winnipeg) 
showing little change in response rate over these cycles. In fact if we exclude the results obtained 
in Toronto, the response rate declined only slightly between Cycle 3 and Cycle 5. We have 
observed that more experienced interviewers tend to be more successful at achieving high 
response rates. The dramatic change in response rates over three cycles experienced by the 
Toronto office may in large part be due to the difficulty in hiring and retaining staff in a city 
that at the time had a booming economy. It is also possible that some of the change is due to 
a change in the population sampled from the Toronto office. 

Preliminary results from eight (January to August 1991) months of data collection for 
Cycle 6 indicate (see Table 2) that it was possible to reverse the trend to lower response rates. 
There were a number of changes made between Cycles 5 and 6, the most important ones being 
a change to monthly data collection and the reassignment of the sample from offices not used 
for data collection in Cycle 6: St. John’s’ sample was transfered to Halifax, Toronto’s to 
Sturgeon Falls and Edmonton’s to Winnipeg. 

During data collection for Cycle 3 it was noted by interviewers that an increasing number 
of calls were answered by answering machines. This raised the concern that respondents might 
use these machines to screen their calls, resulting in higher non-response rates. We are not able 


Table 1 
Response and Non-response Rates (%) by Cycle and Type 


Cycle 
Result eee ee ee ee ee 
1 2 3 4 5 

Household Refusal 6.2 6.2 6.0 Vee 10.3 
Other Household Non-Response 4.4 6.8 6.6 6.4 dod 
Respondent Refusal 43 2.8 133 13} 2.4 
Other Respondent Non-Response 4.8 35 & 4 3.9 4.3 
Special Cycle 2 Non-Response 1.9 


Response 83.4 78.9 82.9 80.7 75.8 
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Table 2 


Response Rates by Cycle and Regional Office 
(results for Cycle 6 are preliminary - indicates offices not conducting interviews) 


Cycle 
Regional Office 
3 4 5 6 

St. John’s 84.1 82.8 90.9 - 
Halifax 84.7 84.1 85.9 82 
Montreal 83.0 79.6 81.2 82 
Sturgeon Falls 76.5 81.1 71.5 71 
Toronto 87.0 75.4 63.0 - 
Winnipeg 84.3 87.0 84.3 89 
Edmonton 83.2 79.4 76.8 - 
Vancouver 15%3 80.2 79.6 82 
Canada 82.9 80.7 75.8 81 
Canada (without Toronto) 82.1 81.8 80.1 - 


Table 3 
Response and Non-response Rates (%) by Type and Contact with Answering Machines 


Did any calls reach an answering machine? 


Cycle 4 Cycle 5 
No Yes No Yes 
Household Refusal 7.18 8.17 10.34 9.74 
Other Household Non-Response 6.45 5.89 7:19 Tal 
Respondent Refusal 5.05 4.55 4.39 3.27 
Other Respondent Non-Response 1.68 2.01 Zo Soke 
Responses 79.64 79.38 75.76 76.44 
Number of Records 10,981 747 16,611 1,715 
(93.6%) (6.4%) (90.6%) (9.4%) 
Table 4 


Response and Non-response Rates (%) by Type and Type of First Contact 


Was the first contact with an answering machine? 


Cycle 4 Cycle 5 
No Yes No ¥es 
Household Refusal 7.24 7.19 10.46 7.90 
Other Household Non-Response 6.46 5.40 1 bi, 8.06 
Respondent Refusal 5.08 3.96 4.38 3.07 
Other Respondent Non-Response 1.70 1.80 2.43 1.92 
Responses 19 Sa 81.65 75.58 79.05 
Number of Records 1 sd Pe 556 17,023 1,303 


(95.3%) (4.7%) (92.9%) (7.1%) 
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to identify those calls that were answered by a machine for cycles 1 to 3, but we are for subse- 
quent cycles and so can analyze to some extent the effect of their use on response rates. Table 3 
compares the response rates for those households for which none of the calls were answered 
by a machine with those for which at least one call was. No important effect of answering 
machines is indicated by this table; however the increase in contacts with answering machines, 
from 6.4% to 9.4% of households, is dramatic (Table 3). Table 4 compares the response rates 
for those households for which the first answered call was answered by a machine with those for 
which it was not. If any effect of answering machines is indicated by this table it is that response 
rates are higher for those households with a first contact by answering machine. There appears 
to be no evidence that the use of answering machines is seriously reducing response rates. 


3.10 Data Capture and Processing 


The data for all five cycles were captured directly into the mini-computers in Statistics 
Canada’s regional offices. Some simple edits to check the validity of data as captured were 
made at the time of capture, but these could in most cases be overridden using special func- 
tions. Following transmission of the raw data to Ottawa an exhaustive set of edits was applied 
to find, and correct if possible, invalid or inconsistent responses. When a response was missing, 
invalid or inconsistent with other responses and the approriate value could not be inferred from 
other responses on the questionnaire an ‘unknown’ code was assigned. Exceptions to this rule 
were three variables needed for weighting purposes: age, sex and number of telephone lines. 
In cases where these variables were missing the questionnaires themselves were consulted to 
assist in the imputation of values. 


3.11 Weighting 


3.11.1 Initial Weights 


Both the Waksberg and ENWB methods of selecting RDD samples yield self-weighting 
samples of residential telephone numbers. The Waksberg method does not provide an estimate 
of this weight, but for GSS weighting purposes it is sufficient to use an initial weight of one 
(1) for telephone numbers in those strata where that method is used. In ENWB strata the initial 
weight is the reciprocal of the probability of selection of the telephone number. This probability 
is simply: 


pactlerai 
100 x Nz 
where: 


n, is the number of telephone numbers selected, and 


Ng is the number of working banks in the frame. 


3.11.2 Non-response Adjustment 


The initial weight is adjusted for non-response using adjustment ‘‘strata’’ based on telephone 
geography. These are typically banks in Waksberg method strata and ACP’s in ENWB strata. 
The initial weights are inflated by the following factor: 

NR ap at! NR 


’ 


NR 
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where: 
Np is the number of responding households in the non-response ‘“‘stratum’’, and 


Nnpr is the corresponding number of non-responding households. 


3.11.3 Telephone Adjustment 


Since households with more than one telephone line have a higher chance of being selected 
by an RDD survey, the initial weight adjusted for non-response (a weight for telephone 
numbers) is further adjusted by dividing by the number of telephone lines for the household 
to yield a household weight. 


3.11.4 Initial Person Weight 


Since only one eligible respondent per household is interviewed, the household weight must 
be adjusted by multiplying by the number of eligible respondents to yield a person weight. 


3.11.5 Poststratification 


At this point populations projected from the census are used as reference totals in the 
poststratification of the person weights, first to the stratum population sizes and then to the 
provincial age-sex populations. (It should be noted that it is only after the first stage of 
poststratification that the weights in Waksberg strata actually sum to a population estimate. 
Until this step they differ from a set of weights based on the inverses of the selection probabilities 
by an unknown constant of proportionality.) These two sets of reference totals are then used 
as the margins for a raking ratio adjustment to the weights. 


4. THEMES AND RESEARCH ISSUES COVERED BY THE GSS 


As indicated above, in order to cover a wide range of social issues, the GSS examines a dif- 
ferent core topic each year for five years and then the topics are repeated. The core topics were 
chosen to fill perceived data gaps in the social statistics program. The five core themes are 
discussed in more detail below. 


4.1 Health 


The core content of the health cycle is directed at providing a range of measures of health 
status, including short and long term disability, the prevalence of common chronic conditions, 
such as high blood pressure or diabetes, and the use of various health care services. In addi- 
tion, data are collected on life-style such as, smoking, drinking, and physical exercise. When 
linked to health status, these data provide information on the barriers (e.g. smoking, drinking) 
and bridges (e.g. physical exercise) to positive health for various population groups. 

For the first GSS health cycle, the add-on focus content was directed at older Canadians 
and covered social networks, support given and received, as well as participation in a range 
of social activities. The sample size for the elderly population was also increased to allow for 
more in-depth analyses. 


4.2 Time Use 


The GSS time use survey consisted of a ‘‘24 hour time budget’’ generally for the day 
preceding the interview. Respondents provided information on each primary activity engaged 
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in during that day, the start time and duration of each activity, and associated information 
on where the activity took place and who was with the respondent at the time (e.g. spouse, 
children, friends, efc.). These data provide information on the frequency with which people 
participate in activities such as paid work, household work, attending cultural events, watching 
television, and the time spent on these activities. 

The survey provides information on how Canadians allocate their time to activities such 
as paid work, housework and other non-market work and leisure activities. The data can be 
used to show constraints that limit a person’s choice of the use of time and how these are 
distributed among different population groups. The inclusion of a battery of questions on 
satisfaction with various dimensions of life allows such measures to be correlated with patterns 
of time use for different population groups. 

The 1986 GSS time use cycle also included a small module on intergenerational social mobility 
that allows for the analysis of movement on an occupational or educational hierarchy between 
the respondent and his or her parents. 

The add-on focus content for the time use cycle was a detailed set of questions on language 
knowledge and use. While focus content is generally expected to be related to and complement 
core content, there was a demand for much more detailed language data than could be included 
in the population census. The information collected included data on language use at various 
stages of life (e.g. first learned, during childhood, at school) and in various settings including 
at home, at work, with friends, watching television, and in dealing with federal agencies. In 
order to allow a more detailed analysis in bilingual regions of the country, sample size was 
also increased in these geographic areas. 


4.3. Personal Risk 


The third GSS cycle was based around the topic of personal risk, including both criminal 
victimizations and accidents. Traditionally, information on these topics has been derived from 
administrative sources, such as police statistics and hospital records. However, these data pro- 
vide very little information about the victim and, in addition, there are many crimes (the GSS 
estimates more than half) and accidents which are not reported to authorities. 

The personal risk survey conducted in early 1988 asked respondents about criminal victimizations 
and accidents that they had experienced during calendar year 1987. Data were also collected 
on several life-style measures, such as alcohol consumption and frequency of night outings 
to allow these to be correlated with criminal victimizations and accidents. For each reported 
crime or accident incident, data were collected on the nature of the incident, the consequences 
in terms of activity restriction, medical attention and financial loss. In addition, respondents 
were asked to report their perceptions of crimes and accidents and about precautions taken 
to prevent these events. 

The add-on focus content for the personal risk cycle was a set of questions on contact with 
the criminal justice system (e.g. police, courts, lawyers) and on the awareness and use of services 
by victims of crime. 


4.4 Education and Work 


While the monthly Labour Force Survey and other labour related surveys provide a wealth 
of information about the labour force, none of the existing surveys provides much information 
on the social aspects of work or the perceived quality of working life. The GSS cycle on educa- 
tion and work, conducted in early 1989, was designed to partially fill this data gap. 
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The survey was developed around three main themes that reflect fundamental changes in 
Canadian society: patterns and trends in work and education; new technologies and human 
resources; and work in the service economy. The themes reflect a range of issues on which more 
information is required. For example, the accelerating rate of technological innovation demands 
detailed knowledge about the utilization of and training for computers. Concerns about the 
effective utilization of the nation’s human capital require a better understanding of the links 
between the labour force and the educational system. We also must anticipate future demands 
on educational institutions, and changing relationships between educational attainment and 
socio-economic outcomes. This round of the GSS also augments existing data sources by 
providing new information about the elderly population as well as some of the socio-economic 
implications of the baby boom generation entering middle age. 

The survey collected a partial work and education history. It also included information on 
technology training and the use of computers, and on future plans for education. Subjective 
information also was sought in the form of a series of questions about satisfaction with retire- 
ment and other dimensions of life, as well as a block of questions on attitudes to science and 
technology. 


4.5 Family and Friends 


The fifth cycle of the GSS was based around issues related to family and friends and was 
completed in early 1990. While the Census and other household surveys provide family-based 
data, changes in family life have resulted in a need for new types of information. One short- 
coming of existing data is that generally they are based on a rather narrow concept of the family, 
in particular a nuclear family of parents and children or perhaps an economic family of related 
individuals living in the same household. This survey looks at the family in a broader context 
and collects information on the extent and nature of kinship networks and related questions 
of patterns of informal help and support among family and friends. 

A second major theme of the survey is a result of the trends in marriage, divorce, and the 
increased frequency of common law unions. Increased numbers of Canadians are living in more 
than one union during their life time. The impact of such changes on family life and children 
is substantial and can best be studied by an analysis of marital and family history data. Such 
data were retrospectively collected in a special Family History Survey conducted in 1984 (Burch 
1985). The GSS family cycle incorporates the collection of these data on a regular basis. Specific 
issues that can be addressed include changing patterns of union formation and dissolution, 
the situation of single parent families, and home leaving patterns of young adults. 

A third but more minor theme of the cycle is concerned with the division of household 
labour. 


5. PROGRAM OUTPUTS 


The GSS results are disseminated in a variety of ways. For each survey there are one or more 
publications that present the results of data analysis with respect to particular social issues and 
the monitoring of conditions and trends. The results of Cycle 1 are reported in Statistics Canada 
(1987) and Stone (1988); the results of Cycle 2 are presented in Harvey et a/. (1991) and Creese 
et al. (1991); and the Cycle 3 results are reported in Sacco and Johnson (1990) and Millar and 
Adams (1991). Publications containing the results of other cycles are in preparation. The gen- 
eral public are made aware of GSS results through the publication of reports in the media which 
are often based on articles published in Canadian Social Trends, a quarterly Statistics Canada 
publication that is targeted to a general audience. 
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A second product is a public use microdata file and associated documentation to enable 
university and other researchers to carry out their own analysis of the data. These data are 
also useful for teaching purposes. Microdata files from the first five survey cycles are now 
available. 

In addition to the product outputs, the GSS program has developed a survey capacity. This 
is not simply a system for data collection and processing, but includes other major components. 
Content research and development, related data specification, analysis of survey and other 
relevant data, dissemination of informative results as well as the development and use, where 
applicable, of improved methods of collection, processing, analysis and dissemination are all 
components of the evolving survey capacity of the GSS group. 


6. FUTURE PLANS 


As the GSS program moves into the second round of surveys, attention has shifted from 
the problems of developing and fielding five new surveys to further building the survey program 
through partnerships with others. The first round of surveys has had a modest success with 
obtaining buy-ins of additional sample and/or focus content. Only Cycle 4 had neither focus 
content nor increased sample size. For the first time, the 1990 survey had provincial 
participation, with the Ontario government funding an increase in sample size. 

A new initiative for the GSS program is an investigation of the potential for expanding the 
scope of the survey to include interviewing a sub-sample of respondents again in future cycles. 
In the short term, this could provide an enriched data set by linking content from different 
cycles. In the longer term, it could serve to provide longitudinal data by interviewing respondents 
on the same topic five years later. A feasibility study was conducted in 1990 and the possibility 
of interviewing a sample of respondents from a previous cycle is now offered to interested 
sponsors. 

The GSS will also continue to undertake a range of more general research and development 
activities. Core content of the first set of cycles will be reviewed and input sought from users 
as to possible improvements for future cycles. While new and alternative survey designs and 
approaches will be considered, any potential changes will have to be balanced against the impact 
on data comparability that is required for the long term goal of monitoring change. In addi- 
tion, the content from the first round of surveys will be reviewed from the point of view of 
consistency and integration across GSS survey cycles and between the GSS and the 1991 Census 
and other household surveys. On-going development of the GSS infrastructure will also con- 
tinue. Consideration was given to changing to monthly data collection (and monthly data 
collection was implemented for Cycle 6) and will be given to supplemental collection methods 
(e.g. mail). Attempts are also being made to shift processing of the survey to a micro-computer 
environment to further improve timeliness. Finally, new procedures, such as computer-assisted 
telephone interviewing, will be considered as these become available as part of a larger Statistics 
Canada survey development program. 

In summary, the GSS Program during the coming years will focus on building on the firm 
foundation that has been established during the first round of surveys. The primary objective 
will continue to be the measurement of social conditions and the gradual development of a 
time series to monitor trends. In addition, flexibility will be maintained in order to quickly 
respond to new and emerging social information needs. 
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GUIDELINES FOR MANUSCRIPTS 


Before having a manuscript typed for submission, please examine a recent issue (Vol. 10, 
No. 2 and onward) of Survey Methodology as a guide and note particularly the following 


points: 

if Layout 

1.1 Manuscripts should be typed on white bond paper of standard size (8% x 11 inch), 
one side only, entirely double spaced with margins of at least 1% inches on all sides. 

1.2 The manuscripts should be divided into numbered sections with suitable verbal titles. 

1.3. The name and address of each author should be given as a footnote on the first page 
of the manuscript. 

1.4 Acknowledgements should appear at the end of the text. 

1.5 Any appendix should be placed after the acknowledgements but before the list of 
references. 

2; Abstract 
The manuscript should begin with an abstract consisting of one paragraph followed 
by three to six key words. Avoid mathematical expressions in the abstract. 

3. Style 

3.1 Avoid footnotes, abbreviations, and acronyms. 

3.2 Mathematical symbols will be italicized unless specified otherwise except for functional 
symbols such as “exp(-)” and “log(-)’’, etc. 

3.3. Short formulae should be left in the text but everything in the text should fit in single 
spacing. Long and important equations should be separated from the text and numbered 
consecutively with arabic numerals on the right if they are to be referred to later. 

3.4 Write fractions in the text using a solidus. 

3.5 Distinguish between ambiguous characters, (e.g., w, w; 0, O, 0; 1, 1). 

3.6 Italics are used for emphasis. Indicate italics by underlining on the manuscript. 

42008 igures and Tables 

4.1 All figures and tables should be numbered consecutively with arabic numerals, with 
titles which are as nearly self explanatory as possible, at the bottom for figures and 
at the top for tables. 

4.2 They should be put on separate pages with an indication of their appropriate place- 
ment in the text. (Normally they should appear near where they are first referred to). 

De References 

5.1 References in the text should be cited with authors’ names and the date of publication. 
If part of a reference is cited, indicate after the reference, e.g., Cochran (1977, p. 164). 

5.2. The list of references at the end of the manuscript should be arranged alphabetically 


and for the same author chronologically. Distinguish publications of the same author 
in the same year by attaching a, b, c to the year of publication. Journal titles should 
not be abbreviated. Follow the same format used in recent issues. 
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